IN CASE YOU MISSED IT!

Bigdata, machinelearning, deeplearning, bigdata & much more…

BigData News Thursday, June 14

Understanding Stream Processing – Dzone Refcardz

  • A simple map stage transforms items with a stateless function; a more complex windowed group-and-aggregate stage groups events by key in an infinite stream and calculates an aggregated value over a sliding window.
  • When counting the records in the stream, for example, you have to maintain the current count.
  • This code samples shows both stateless and stateful transformations: – – In the most general case, the state of stateful transformations is affected by all the records observed in the stream, and all the ingested records are involved in the computation.
  • This fact can be leveraged for easy parallelization by partitioning the data stream on the grouping key and letting independent threads/processes/machines handle records with different keys.
  • This example processes a stream of text snippets (tweets or anything else) by first splitting them into individual words and then performing a windowed group-and-aggregate operation.

Tags:

Understanding Stream Processing – Dzone Refcardz

Introduction to Deep Learning: Machine Learning vs. Deep Learning Video

  • Deep learning and machine learning both offer ways to train models and classify data.
  • To have a computer do classification using a standard machine learning approach, wed manually select the relevant features of an image, such as edges or corners, in order to train the machine learning model.
  • When choosing between machine learning and deep learning, you should ask yourself whether you have a high-performance GPU and lots of labeled data.
  • But in a deep learning model, you need a large amount of data, which means the model can take a long time to train.
  • By using deep learning as a feature extractor and machine learning to classify the features, you can get an accurate and flexible model.

Tags:

Introduction to Deep Learning: Machine Learning vs. Deep Learning Video

Big Oil harnesses power of data analysis to ensure survival

  • $60bn – – Estimated value of potential annual productivity gains for European oil and gas – majors – – The powerful software has similar benefits in optimising the operation of – existing fields, as well as downstream facilities, such as pipelines and – refineries.
  • Uses range from managing reservoir pressure to maximise oil and gas – flows, to utilising artificial intelligence to predict the need for maintenance – work.
  • If you can avoid drilling a well or avoid a maintenance shutdown, you have – already made a huge return on your investment in the technology, says oil and gas groups are making similar investments.Total agreed adeal with – Google in April to jointly develop artificial intelligence for data analysis…
  • Lydia Rainforth, analyst at Barclays, says that use of digital technologies to – improve productivity is set to transform the big oil companies over the coming – decade.
  • This is simply not sustainable and it is clear to us that a new way of working – is required, says Ms Rainforth, who estimates the value of the potential – productivity gains for European oil and gas majors at an annual $60bn.

Tags:

Big Oil harnesses power of data analysis to ensure survival

Top Big Data Courses

The Ultimate Hands-On Hadoop - Tame your Big Data! (31,889 students enrolled)

By Sundog Education by Frank Kane
  • Design distributed systems that manage "big data" using Hadoop and related technologies.
  • Use HDFS and MapReduce for storing and analyzing data at scale.
  • Use Pig and Spark to create scripts to process data on a Hadoop cluster in more complex ways.
  • Analyze relational data using Hive and MySQL
  • Analyze non-relational data using HBase, Cassandra, and MongoDB
  • Query data interactively with Drill, Phoenix, and Presto
  • Choose an appropriate data storage technology for your application
  • Understand how Hadoop clusters are managed by YARN, Tez, Mesos, Zookeeper, Zeppelin, Hue, and Oozie.
  • Publish data to your Hadoop cluster using Kafka, Sqoop, and Flume
  • Consume streaming data using Spark Streaming, Flink, and Storm

Learn more.


Taming Big Data with MapReduce and Hadoop - Hands On! (13,894 students enrolled)

By Sundog Education by Frank Kane
  • Understand how MapReduce can be used to analyze big data sets
  • Write your own MapReduce jobs using Python and MRJob
  • Run MapReduce jobs on Hadoop clusters using Amazon Elastic MapReduce
  • Chain MapReduce jobs together to analyze more complex problems
  • Analyze social network data using MapReduce
  • Analyze movie ratings data using MapReduce and produce movie recommendations with it.
  • Understand other Hadoop-based technologies, including Hive, Pig, and Spark
  • Understand what Hadoop is for, and how it works

Learn more.