Kishore

Hii

Kishore

😌

Kishore

How to convert string to time stamp in dataframe My column value is 2013-07-25 00:00:00.0

Grigory

How to convert string to time stamp in dataframe My column value is 2013-07-25 00:00:00.0

1. you can also search by key words in google such questions 2. https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala

Anonymous

https://www.oreilly.com/ideas/handling-real-time-data-operations-in-the-enterprise

Ranganath

Hi , I have a table in hive whose delimiter is '^' and data file also loaded with '^' delimiter but now we got a requirement to change the delimiter to '^A' ....I changed the delimiter in table but the data now is being displayed as NULL as data in file having different delimiter ...how can I change delimiter at file level also ... Or do I need to create staging table for all table but this will be a problem as I have 20 tables ...

Anonymous

Could help me starting in data engineering, best plan?

KrivdaTheTriewe

Could help me starting in data engineering, best plan?

yandex course of big data engineering on coursera

Pedro

Hey people !

Alexander

Hi!

Mikhail

Hey people !

Hola, que tal?

Pedro

How is going ?

Grigory

How is going ?

not much, whats up u?

Pedro

not much, whats up u?

cool i'm fine trying to solve a problem with fixed-width row to print in files to send to a legacy system to consume and then back to HDFS

Gurmohit Singh

Hello :)

Bevilaqua

Some here who knows Google Dataprep tool?

Pedro

Some here who knows Google Dataprep tool?

What's the question ?

Anonymous

Dear experts, I am new on grafana to make a monitoring dashboard. Ive connect my data source with CloudWatch. And monitoring an AWS/EC2 instance But only 14 metrics were available on grafana. I need most of all metrics of ec2 instance. Need help thanks

Pedro

Dear experts, I am new on grafana to make a monitoring dashboard. Ive connect my data source with CloudWatch. And monitoring an AWS/EC2 instance But only 14 metrics were available on grafana. I need most of all metrics of ec2 instance. Need help thanks

This is done via parameters on CloudWatch and how it populates the dashboard

Anonymous

Hello everyone... I have testing 3.8 years of experience.. Im very much intrested to work as data engineer .. Can someone help me with the skills required to restart my career as data engineer... Please let me know If there are any training institutions which can help to upskill my self in Hyderabad... Help me with ur ideas and suggestions

Saipul

Hello everyone... I have testing 3.8 years of experience.. Im very much intrested to work as data engineer .. Can someone help me with the skills required to restart my career as data engineer... Please let me know If there are any training institutions which can help to upskill my self in Hyderabad... Help me with ur ideas and suggestions

so currently now you are work as Software tester right

Saipul

maybe you can take course on dataquest

Saipul

choose data engineer track

Saipul

online course

Pedro

Just study what's does a data engineer. There is some great blog posts about that over the internet

Pedro

Medium is a good starting point

Sam

Hello, Could anybody are familiar with airflow? I got stuck with hive jdbc connection. Could you please take a look at the problem through this link? https://stackoverflow.com/questions/55134669/cannot-modify-mapred-job-name-at-runtime-it-is-not-in-list-of-params-that-are-a

Pedro

Airflow and jdbc ?

Pedro

Ooh

Pedro

I see

Pedro

Hive and jdbc

Pedro

Sorry never worked with those

Sam

yeap, hive jdbc.

Ashhadul

Hi Anyone from Hyderabad, India Interested to learn data science

Anonymous

Yes

Anonymous

I m interested

KrivdaTheTriewe

https://www.youtube.com/watch?v=K6oZuB8_dU8&feature=youtu.be

Anonymous

Hi Anyone from Hyderabad, India Interested to learn data science

Hi Islam, I am from Hyderabad, I am interested in learning.

Saipul

Oleksandr

No Java, Scala and JS? I don’t think so.

Oleksandr

Also nowadays majority goes. Python -> AWS/Azure. Why bother learning stuff in between.

Pedro

The best language for the job is the right thing. The best languages for some cases are the one which has middle set of features and some sort of simplicity, mainly those for systems programming (Go, Rust, Elixir are good ones)

Pedro

This mainstream learning path is all wrong and really passed away out of reality

Pedro

A good engineer, concerning big data, will employ the language which best fits the problem and not what some dinosaurs is telling as single truth

A

Visual Studio Code Lint displays errors about imports that dont seem to harm the scripts I run usin same VS Code. Is it a known issue?

Ashhadul

“Generic Review Scraper for any Android App” by Ashhadul Islam https://link.medium.com/vjLOgtZzvW

Ashhadul

“Generic Review Scraper for any Android App” by Ashhadul Islam https://link.medium.com/vjLOgtZzvW

This is a medium post about a free web application that scrapes reviews of any android app from google play store. Do take a look. Steps to use the app are also mentioned.

B

what do you guys think about serverless data lake?

B

mainly on AWS

Grigory

@voreh hey; what do you mean by that? what would be a serverless component?

Grigory

where is the data

Grigory

or you talk about that AWS blogpost?

Grigory

that’s fine; if you have enough will to create your own serverless API around any data storage - why not to do it; sounds okay

B

@pomadchin well, install all softwares and keep them updated, calculate the amount of RAM, SSD, sound too much for me when you can use a serverless data lake for your applications

Grigory

its only for the API part; AWS proposes to store data in their s3 storage

Grigory

not sure how is it 'serverless'

Grigory

they just used this bazzword IMO

Grigory

they also have smth pre done https://aws.amazon.com/ru/solutions/data-lake-solution/

B

well, i mean all the resources together to create a serverless infrasctrucutre, like the link you send it, (s3, lambda, kinesis, athena) without create an EC2 instance

Grigory

well I guess by serverless they meant (and you mean) basically to lock on some vendor (aws in our case) software (SAAS)

Alexander

what do you guys think about serverless data lake?

Well under serverless you mean services without launching certain EC2 in your account (only API calls). You can start with S3 (storage), Lambda, Glue (ETL and DC), Athena (SQL engine), Kinesis, ALB, etc. building a data platform (we successfully leveraged them on one of the projects) covering data ingestion, transformation, discovery and DG, etc. but probably you'll come to using EMR, EKS, etc. also for further steps.

Grigory

++

Ashhadul

https://mysentimeter.herokuapp.com/senti/ I have built a simple generic django application that enables any end-user to perform text classification. User guide: https://medium.com/@ashhadulislam/generic-sentiment-analysis-on-cloud-5456131ba461 Do use the same and let me know

Sparsh

https://link.medium.com/udTjR9CklX

K.K

Plz anyone can suggest me regarding the preparation of cloudera certification for data engineering..CCP DE 575.

K.K

Any body have some idea then plz ! Honourable members guide me..😔

Saipul

Hello guys, are there use MapR in your company ?

Ashhadul

Guys if you want to learn how to build a chatbot from scratch, I take one hour session in which I teach you to build a chatbot. It's not free, it's 500 rupees per person, per session. If you are interested to learn, please ping me. Pre requisite: hands: experience with python coding

Rakz

Hi all, is there any implementation of scala operator in Apache airflow for scheduling. Is there any scheduler available for scala?, not looking for a cron though

Grigory

Hi all, is there any implementation of scala operator in Apache airflow for scheduling. Is there any scheduler available for scala?, not looking for a cron though

what do you mean by the Scala operator; what should it do?

Rakz

I’m looking for an option to run scala code via Airflow or any suggestions on how to schedule a scala code without using crontab

Grigory

I’m looking for an option to run scala code via Airflow or any suggestions on how to schedule a scala code without using crontab

It doesn’t matter what code to run; you can use BashOperator

Grigory

There is nothing special in a Scala code jar

Rakz

Got it, this is because Airflow uses python. I wanted to check if any code samples available for scala based scheduler