Hii

😌

How to convert string to time stamp in dataframe My column value is 2013-07-25 00:00:00.0

How to convert string to time stamp in dataframe My column value is 2013-07-25 00:00:00.0

1. you can also search by key words in google such questions 2. https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala

https://www.oreilly.com/ideas/handling-real-time-data-operations-in-the-enterprise

Hi , I have a table in hive whose delimiter is '^' and data file also loaded with '^' delimiter but now we got a requirement to change the delimiter to '^A' ....I changed the delimiter in table but the data now is being displayed as NULL as data in file having different delimiter ...how can I change delimiter at file level also ... Or do I need to create staging table for all table but this will be a problem as I have 20 tables ...

Could help me starting in data engineering, best plan?

Could help me starting in data engineering, best plan?

yandex course of big data engineering on coursera

Hey people !

Hi!

Hey people !

Hola, que tal?

How is going ?

How is going ?

not much, whats up u?

not much, whats up u?

cool i'm fine trying to solve a problem with fixed-width row to print in files to send to a legacy system to consume and then back to HDFS

Hello :)

Some here who knows Google Dataprep tool?

Some here who knows Google Dataprep tool?

What's the question ?

Dear experts, I am new on grafana to make a monitoring dashboard. Ive connect my data source with CloudWatch. And monitoring an AWS/EC2 instance But only 14 metrics were available on grafana. I need most of all metrics of ec2 instance. Need help thanks

Dear experts, I am new on grafana to make a monitoring dashboard. Ive connect my data source with CloudWatch. And monitoring an AWS/EC2 instance But only 14 metrics were available on grafana. I need most of all metrics of ec2 instance. Need help thanks

This is done via parameters on CloudWatch and how it populates the dashboard

Hello everyone... I have testing 3.8 years of experience.. Im very much intrested to work as data engineer .. Can someone help me with the skills required to restart my career as data engineer... Please let me know If there are any training institutions which can help to upskill my self in Hyderabad... Help me with ur ideas and suggestions

Hello everyone... I have testing 3.8 years of experience.. Im very much intrested to work as data engineer .. Can someone help me with the skills required to restart my career as data engineer... Please let me know If there are any training institutions which can help to upskill my self in Hyderabad... Help me with ur ideas and suggestions

so currently now you are work as Software tester right

maybe you can take course on dataquest

choose data engineer track

online course

Just study what's does a data engineer. There is some great blog posts about that over the internet

Medium is a good starting point

Hello, Could anybody are familiar with airflow? I got stuck with hive jdbc connection. Could you please take a look at the problem through this link? https://stackoverflow.com/questions/55134669/cannot-modify-mapred-job-name-at-runtime-it-is-not-in-list-of-params-that-are-a

Airflow and jdbc ?

Ooh

I see

Hive and jdbc

Sorry never worked with those

yeap, hive jdbc.

Hi Anyone from Hyderabad, India Interested to learn data science

Yes

I m interested

https://www.youtube.com/watch?v=K6oZuB8_dU8&feature=youtu.be

Hi Anyone from Hyderabad, India Interested to learn data science

Hi Islam, I am from Hyderabad, I am interested in learning.

No Java, Scala and JS? I don’t think so.

Also nowadays majority goes. Python -> AWS/Azure. Why bother learning stuff in between.

The best language for the job is the right thing. The best languages for some cases are the one which has middle set of features and some sort of simplicity, mainly those for systems programming (Go, Rust, Elixir are good ones)

This mainstream learning path is all wrong and really passed away out of reality

A good engineer, concerning big data, will employ the language which best fits the problem and not what some dinosaurs is telling as single truth

Visual Studio Code Lint displays errors about imports that dont seem to harm the scripts I run usin same VS Code. Is it a known issue?

“Generic Review Scraper for any Android App” by Ashhadul Islam https://link.medium.com/vjLOgtZzvW

“Generic Review Scraper for any Android App” by Ashhadul Islam https://link.medium.com/vjLOgtZzvW

This is a medium post about a free web application that scrapes reviews of any android app from google play store. Do take a look. Steps to use the app are also mentioned.

what do you guys think about serverless data lake?

mainly on AWS

@voreh hey; what do you mean by that? what would be a serverless component?

where is the data

or you talk about that AWS blogpost?

that’s fine; if you have enough will to create your own serverless API around any data storage - why not to do it; sounds okay

@pomadchin well, install all softwares and keep them updated, calculate the amount of RAM, SSD, sound too much for me when you can use a serverless data lake for your applications

its only for the API part; AWS proposes to store data in their s3 storage

not sure how is it 'serverless'

they just used this bazzword IMO

they also have smth pre done https://aws.amazon.com/ru/solutions/data-lake-solution/

well, i mean all the resources together to create a serverless infrasctrucutre, like the link you send it, (s3, lambda, kinesis, athena) without create an EC2 instance

well I guess by serverless they meant (and you mean) basically to lock on some vendor (aws in our case) software (SAAS)

what do you guys think about serverless data lake?

Well under serverless you mean services without launching certain EC2 in your account (only API calls). You can start with S3 (storage), Lambda, Glue (ETL and DC), Athena (SQL engine), Kinesis, ALB, etc. building a data platform (we successfully leveraged them on one of the projects) covering data ingestion, transformation, discovery and DG, etc. but probably you'll come to using EMR, EKS, etc. also for further steps.

++

https://mysentimeter.herokuapp.com/senti/ I have built a simple generic django application that enables any end-user to perform text classification. User guide: https://medium.com/@ashhadulislam/generic-sentiment-analysis-on-cloud-5456131ba461 Do use the same and let me know

https://link.medium.com/udTjR9CklX

Plz anyone can suggest me regarding the preparation of cloudera certification for data engineering..CCP DE 575.

Any body have some idea then plz ! Honourable members guide me..😔

Hello guys, are there use MapR in your company ?

Guys if you want to learn how to build a chatbot from scratch, I take one hour session in which I teach you to build a chatbot. It's not free, it's 500 rupees per person, per session. If you are interested to learn, please ping me. Pre requisite: hands: experience with python coding

Hi all, is there any implementation of scala operator in Apache airflow for scheduling. Is there any scheduler available for scala?, not looking for a cron though

Hi all, is there any implementation of scala operator in Apache airflow for scheduling. Is there any scheduler available for scala?, not looking for a cron though

what do you mean by the Scala operator; what should it do?

I’m looking for an option to run scala code via Airflow or any suggestions on how to schedule a scala code without using crontab

I’m looking for an option to run scala code via Airflow or any suggestions on how to schedule a scala code without using crontab

It doesn’t matter what code to run; you can use BashOperator

There is nothing special in a Scala code jar

Got it, this is because Airflow uses python. I wanted to check if any code samples available for scala based scheduler

You mean to write dags in Scala?

You mean to write dags in Scala?

Yeah

only python

only python

Okay thanks for your suggestion

Got it, this is because Airflow uses python. I wanted to check if any code samples available for scala based scheduler

One example:Using Airflow to Manage Talend ETL Jobs Link: https://dzone.com/articles/airflow-to-manage-talend-etl-jobs

One example:Using Airflow to Manage Talend ETL Jobs Link: https://dzone.com/articles/airflow-to-manage-talend-etl-jobs

Good suggestion but my working environment is gcloud. Client will not prefer to buy out licenses or install external tools due to data privacy.