The accuracy & nature of answers you get on large data sets can be completely different from what you see on small samples. Big data provides a competitive advantage. For the web data sets you describe, it turns out that having 10x the amount of data allows you to automatically discover patterns that would be impossible with smaller samples (think Signal to Noise). The deeper into demographic slices you want to dive, the more data you will need to get the same accuracy.

It's a fancy term for essentially large volumes of data. Data so massive that traditional relational databases or data processing applications cannot handle it (either at all, or not in acceptable response times).

Every day, we create 2.5 Quintilian bytes of data — so much that 90% of the data in the world today has been created in the last two years alone. This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few. This data is big data.

I think the main reasons for the obsession with Big Data are the following :

1. Information Asymmetry: Storage gets cheaper every day, and if you have more data you can make better decisions than competitors.
2. The whole is greater than the sum of its parts - many data sets combined tell you more than they do separately
3. Game changing advancements in machine learning are coming from leveraging Big Data

What would you go for mobile app development or big data, if you want to be an entrepreneur?

See doesn't matter you missed app development buzz but tell me how you know missed… Revolution has just began in Mobile… So chill and coming to your point to BigData or Mobile App Development.. Doesn't matter really if you have strong idea to address… What you can do start mobile app idea build a strong user base with huge data and then with that data perform your Big Data operations on it to get insights of your users..

One more thing just because BigData is creating hype doesn't mean you will be successful or same applies for mobile app development. When Facebook was build it was build on basics of web fundamentals like php.. So you cant say you missed buzz on php..

Stop thinking about what people learn or create hype about. Just go by your mind expertise in one field.. May be you may create a next Big Thing.

Which way should you go considering you want to develop mobile-based apps based on big data: Node.js or Scala?

- You're comparing apples and oranges here. Comparing languages would work: "JavaScript vs Scala", or you could compare frameworks: "node.js vs play".  But it's not really possible to compare a language to a framework.

If you'll be learning Scala and Spark soon, then run with that. Node.js (and io.js) is worth learning in its own right, but you won't need it, it's just a way to run server-side JavaScript.

But given Scala, there are other frameworks you could use, such as Play, Spray, Scalatra and Lift. The webpages will still use JavaScript, but you avoid the pain of running multiple backend servers.

- You should JavaScript in your list. So, it makes sense to learn it, if you are interested in maintaining the ability to fast slap together some kind of prototype, leave it to the engineers and move on. This will work.

On the other hand, Scala has a lot of future, so why not learn it. When you are good with functional programming, you'll be a different person. You won't be listing Python and Hadoop among your skillset. They are things of the past; this is 2015 now, and the coding patterns are becoming very different from what we had in that old age, 10 years ago.

What is going to be the next big thing in softwares and information technology? Is it going to be the mobile apps, big data, IoT or something else?

It is going to be big data or mobile. Mobile application development is already the most desired as the world goes mobile and it will keep on growing. Big data hadoop is a concept which was adopted by giants like google and yahoo years back but there was no name for it. Since now the amount of data world is producing on a daily basis it is becoming a challenge for these big giants to find better mechanism to store it, so the big data has a name now. But there is no doubt big data engineers are going to be highly paid engineers.

Why do people think big data is so important?

In simple words, big data is structured/unstructured data and in large amounts. There is no standard format in the data you have.

The important point is, not everyone has a big data problem. Sometimes, they just create big data problems.

Coming to your question, big data has two components to it: 1) Storage 2) Analyzing. What will help us is analysis.

There are lots of things you can learn from a given data. And whatever you learn from it can be translated into business models and improvisations. Big data analytics helps to spot certain patterns and behavior which can be introspected further to understand the causes.

For example, you are analyzing data of a manufacturing process. Through your analysis, you observe the bottleneck in the process. Now, with this insight you can make sure the bottleneck is eliminated.

Another scenario is that of predictive analytics. From the past and present data, you try to predict the future of certain things of interest.

In these and many other ways, big data analytics helps to give you a better insight about many behaviors in lesser time and helps you to act on them swiftly.

Sample of the Applications

Big data includes problems that involve such large data sets and solutions that require a complex connecting the dots. You can see such things everywhere.

1. Quora and Facebook use Big data tools to understand more about you and provide you with a feed that you in theory should find it interesting. The fact that the feed is not interesting should show how hard the problem is.

2. Credit card companies analyze millions of transactions to find patterns of fraud. Maybe if you bought pepsi on the card followed by a big ticket purchase, it could be a fraudster?

3. My cousin works for a Big Data startup that analyzes weather data to help farmers sow the right seeds at the right time. The startup got acquired by Monsanto for big $$.

4. A friend of mine works for a Big Data startup that analyzes customer behavior in real time to alert retailers on when they should stock up stuff.

Closing

Many classic "big data" applications don't actually involve all that much data - police departments that use data mining on police reports to come up with crime heat maps and such things are operating on a few hundred megabytes (at most) of police report location and crime type data in their cities. The thing that made these "big data" applications was the mining and unusual analysis of previously unused data.

Also, many apps that have huge amounts of transnational lookup data, but aren't doing mining or analytics on the data, are not what I'd call big data applications.  Processing a purchase using your credit card isn't a big data app, even though the transaction may interact with some very large databases.