Find Jobs
Hire Freelancers

Configure HDFS and Spark on DC/OS Cluster

€18-36 EUR / hour

Closed
Posted over 5 years ago

€18-36 EUR / hour

We are running DC/OS Cluster on AWS, and manage it using Terraform. We have tried to deploy HDFS on DC/OS, but we still can't connect from locat to cluster's HDFS, and also can't consume the data from HDFS/ S3 using Spark, due to error on the miss of hadoop client. HDFS Cluster and Spark are set up and show healthy states. Client's [login to view URL] and [login to view URL] are also downloaded and placed under local hadoop installation, but. 1) 'hdfs dfs -ls /' lead to error 2) spark complains on problem to find hadop / hdfs Here are the links on the very same problems, but we haven't managed to resolved them. So we're looking for expert help that had already experience on running HDFS / DCOS, so point out where to change. Thanks [login to view URL] [login to view URL] Please answer for below questions when apply to job 1. Do you have experience with DC/OS installs and maintenance? 2. Do you ever configure DC/OS HDFS and DC/OS Spark 3. Have you managed custom Spark deployment (--packages, s3://) 4. Do you have suggestions to make this project run successfully? 5. What questions do you have about the project?
Project ID: 18324290

About the project

10 proposals
Remote project
Active 5 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
10 freelancers are bidding on average €30 EUR/hour for this job
User Avatar
Hello, I can Configure HDFS and Spark on your dc cluster. Please let me know if you are interested so we can discuss regarding the project over chat. Waiting for your reply. Thank you
€36 EUR in 40 days
4.8 (25 reviews)
4.7
4.7
User Avatar
I am working as a software developer with 10+ years of experience. Industry experience includes Big Data, Backend, DevOps (AWS and Azure) and etc. I did my MTECH in the Department of Computer Science Engineering, Indian Institute of Technology (IIT) , Guwahati, India. Some of the skills are mentioned below. Amazon Web Services: EC2, Load Balancers, RDS, Elastic Beanstalk, Athena, Glue, EMR IAM, VPC, Security Groups, Cloud Formation, Lambda, S3, CLoudFront, Redshift, Route 53, Cloud Watch, Kinesis, SNS, SQS, API Gateway and etc. Azure: VMs, SQL Databases, Storage and etc. Programming Languages: C, C++, Java, Go, Python, Bash Big Data Technologies: Hadoop, Spark, Hive, HDFS, HBase, Kafka, Flume, Memsql, Aerospike, Redis and etc. Miscellaneous: Algorithms, Data Structures, Distributed Systems, Operating Systems Databases: Oracle, MySQL, SQL Server Operating Systems: Windows, Linux, Ubuntu and etc. Code Deployments: Puppet, Docker, Airflow
€30 EUR in 15 days
4.9 (28 reviews)
4.8
4.8
User Avatar
Hello I am working in Bigdata/Hadoop technologies for years. I worked in Hadoop, Mapreduce, Spark (Streaming & MLLib), Kafka, ELK Stack, Cassandra, MongoDB, Postgre. I implemented above using Java Scala and Python. Have worked Shell, Perl and Python Can we talk further on this? Thank you!
€30 EUR in 40 days
4.9 (22 reviews)
4.9
4.9
User Avatar
HI, I have more than 4+ years of experience in hadoop technologies like HDFS, MapReduce, Spark, Sqoop etc. Contact me for more detail
€33 EUR in 40 days
4.6 (7 reviews)
3.7
3.7
User Avatar
Hi, I have 7 years of experience and working on hadoop, spark, nosql, java, BI tools(tableau, powerbi), cloud(Amazon, Google, Microsoft Azure)... Done end to end data warehouse management projects on aws cloud with hadoop, hive, spark and presodb. Worked on multiple etl project like Kafka, nifi, flume, mapreduce, spark with XML/JSON., Cassandra, mongodb, hbase, redis, oracle, sap hana, ASE.... Many more. Let's discuss the required things in detail. I am committed to work done and strong in issue resolving as well. Thanks
€20 EUR in 40 days
4.9 (3 reviews)
2.0
2.0
User Avatar
I can help you with this project: - Well Experienced in Big Data (Hadoop | Kafka | Spark | NoSQL | Cloud) Administration and Platforms to Accommodate the Expanding Business Needs. - Well Experienced in Different Vendors (Hortonworks, Cloudera, MapR, etc) of Big Data and Cloud. - Big Data Capacity Planning, Tuning, Solution Architecting, Issue Resolution, and Security. - Big Data Analytics Solution Architecture, and its Implementation. Answers: 1. Do you have experience with DC/OS installs and maintenance? Yes 2. Do you ever configure DC/OS HDFS and DC/OS Spark Yes 3. Have you managed custom Spark deployment (--packages, s3://) Yes 4. Do you have suggestions to make this project run successfully? Once I get more insight into the use case, I can guide accordingly 5. What questions do you have about the project? Once I get more insight into the use case, I might have more questions Thanks, JJ
€30 EUR in 40 days
0.0 (0 reviews)
0.0
0.0
User Avatar
Hello, Hope you are doing absolutely great. Instead of writing Proposal I would like to approach you directly for this job, as we have been working on the AWS since last 3 years. Let's take this forward as it looks like you need of an AWS expert to work on running HDFS / DCOS, so point out where to change. Hours of work: 40 Hr/week We have a working experience in node.js, MongoDB, Express JS, NoSQL Couch & Mongo, PostgreSQL, webRTC, MySQL, PHP, System Admin, frontend and backend development, load balancing, Video Broadcasting, Video Services, Amazon Web Services, Cloud Computing, Linux, Network Administration, System Admin, Database Management. So, here I’m confident that we will able to work on your project. Looking for to connect with you in chat. Best Regards, Akash!
€20 EUR in 40 days
0.0 (0 reviews)
0.0
0.0

About the client

Flag of NETHERLANDS
Netherlands
0.0
0
Member since Dec 8, 2018

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.