Welcome!

Python Authors: Matt Davis, Jyoti Bansal, Pat Romanski, Donald Meyer, Liz McMillan

Related Topics: @CloudExpo, Linux Containers, @BigDataExpo, @ThingsExpo, @DevOpsSummit

@CloudExpo: Article

In-Stream Processing | @CloudExpo @robinAKAroblimo #BigData #AI #BI #DX

We are still getting sales reports and other information we need to run our business long after the fact

Most of us have moved our web and e-commerce operations to the cloud, but we are still getting sales reports and other information we need to run our business long after the fact. We sell a hamburger on Tuesday, you might say, but don't know if we made money selling it until Friday. That's because we still rely on Batch processing, where we generate orders, reports, and other management-useful pieces of data when it's most convenient for the IT department to process them, rather than in real time. That was fine when horse-drawn wagons made our deliveries, but it is far too slow for today's world, where stock prices and other bits of information circle the world (literally) at the speed of light. It's time to move to In-Stream Processing. You can't - and shouldn't - keep putting it off.

[Figure 1, courtesy of the Grid Dynamics Blog]

This diagram may look complicated at first, but if you trace the lines it will soon become clear. The only thing that might throw some managers for a loop is the "Data Science" box at the bottom. This term may seem intimidating, but in real life it's just a method of deciding what data is most important to extract from the data stream as it flows by, and how it should best be displayed to the business people who are its end users. Some say, "Data science is just a sexed-up term for statistics." Perhaps. But title aside, displaying the dynamic information needed to run a business - and only that information - in real-time is what In-Stream Processing is all about.

Data Nuggets in the Stream
A retailer may do 100,000 POS credit card transactions per day. That's nice to know, and it's nice to have a dashboard-type display that shows them in real-time, and shows upward and downward trends minute by minute so that cashiers can be assigned with maximum efficiency. In-Stream Processing can do this, and with the right output devices, can even trigger a storewide PA announcement that automatically says, "All cashiers to the front, please." Opening more registers as soon as checkout volume starts to trend upwards makes customers happy. Even the announcement makes them happy, because it shows that management cares about them.

And buried in those 100,000 credit card transactions, there is one that is an attempt to use a stolen card. Obviously, that transaction will be declined. But then what? In the era of batch processing, an automated alert might get sent to management or loss control hours or days after the miscreant was gone. Perhaps the cashier called security, perhaps he didn't. After all, he had other customers in line, and when he told the person with the hot credit card that it was declined, the criminal probably bolted, and the cashier didn't give chase because it was probably against store policy - and he had other things to worry about, anyway.

Now let's fast-forward to tomorrow, when we have In-Stream Processing up and running. The second - literally the second - the hot credit card transaction takes place, both the store's loss control people and management at HQ learn about it. Store security can head for the appropriate POS before the bogus customer is informed that "his" credit card is no good. Indeed, notifying the cashier, and therefore the bogus customer, that the card has been declined can be automatically delayed until store security personnel acknowledge receipt of the alert, which gives them a far greater chance to detain the criminal than they had in the bad old days before they had In-Stream Processing as an anti-theft weapon.

Predicting production failures with In-Stream Processing
Now, let's turn our sights from the POS system to the conveyors in our packing and shipping facility. Whoops! It seems the drive motor for belt number five is suddenly running at 40 degrees C instead of its typical 30C - 32C. If we don't learn about this for a day or two, chances are we'll have burnt-out motor to replace, and we've had a packing line down for a number of hours.

With In-Stream Processing, we can see temperature sensor output changes instantly, and even have alerts set to go off if they move more than X degrees outside their normal operating range. We can do the same thing with strain gauges and many other measurement devices. We can even check the number of boxes getting packed on each line, with alerts set for changes beyond our expectations so that a manager can check that line to make sure everything is okay both with the workers and with their equipment.

The sooner a potential problem is detected, the better our chance of solving it before it becomes a major, costly problem. This applies to almost every aspect of our business, including a sudden spate of customer calls about problems with a particular product.

Keeping irritated customers from becoming angry customers
An exciting recent development in call center management, which couldn't have happened before big data and the cloud made immense data processing power and data storage available at low (and ever-dropping) prices is Emotion Analysis. It's no great trick for a human to tell whether a caller is happy, inquisitive, upset or downright angry. But this is a recent trick for computers, and is just now starting to become a practical business application.

The idea is that as soon as the system detects unhappiness beyond a preset level in a customer's voice, the call is automatically diverted to a supervisor or 2nd-level support person. In theory, a first-tier support person should be able to detect that unhappiness and call for help, but as you know from your own experience calling businesses for help, people in the first support tier may not be capable of recognizing unhappiness even if - it often seems - you bluntly say, "I'm unhappy. Please get your supervisor."

Think what a pleasant surprise it would be if a fresh voice came on the line and said, "This is Ron, in support management. It sounds like you're upset. I don't blame you. Let's see what we can do to solve your problem!" We're not using a chatbot in this scenario - yet. It may not be long until we have good enough AI (or at least pseudo-AI), and good enough voice recognition to replace human customer service workers, but right now nothing beats a knowledgeable employee with the authority to actually make things right for a customer who has gotten a defective product or poor service of some sort.

But In-Stream Processing, running Emotion Analysis, can certainly help us hook our unhappy customer up with the person who can make her problems go away - and do it right now, not a week from Sunday.

New uses for In-Stream Processing are cropping up all the time
Indeed, Emotion Analysis and dozens of other applications that are new or are still being developed all depend on In-Stream Analysis, because they all rely on real-time or near-real-time processing, not processing that happens someday. Program trading is a great example. If you're doing high-frequency stock buying and selling, making hundreds or thousands of trades per minute (or even per second in some cases), you must be able to process data and make decisions - or have a program that makes decisions - fast enough that the length of your connection to the stock exchange can make a noticeable difference in your profits -- which is pretty darn fast.

-------------------------

With a little thought, you can almost certainly think of at least a few ways In-Stream Processing can benefit your business. If not, throw the idea out to your fellow executives. The chances are 100%, more or less, that within a week they'll think of at least a few ways In-Stream Processing can increase your profits.

More Stories By Robin Miller

Robin “Roblimo” Miller is a long-time IT journalist known for his work on Slashdot, Linux.com, and other sites covering software so new that its edges haven’t even started to bleed. Nowadays, he writes for FOSSForce.com and works as an editorial consultant and blog editor for Grid Dynamics, “the engineering IT services company known for transformative, mission-critical cloud solutions for retail, finance and technology sectors.”

@ThingsExpo Stories
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs ofte...
SYS-CON Events announced today that Systena America will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Systena Group has been in business for various software development and verification in Japan, US, ASEAN, and China by utilizing the knowledge we gained from all types of device development for various industries including smartphones (Android/iOS), wireless communication, security technology and IoT serv...
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real r...
The 21st International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
SYS-CON Events announced today that Hitachi Data Systems, a wholly owned subsidiary of Hitachi LTD., will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City. Hitachi Data Systems (HDS) will be featuring the Hitachi Content Platform (HCP) portfolio. This is the industry’s only offering that allows organizations to bring together object storage, file sync and share, cloud storage gateways, and sophisticated search and...
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...
SYS-CON Events announced today that Carbonite will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Carbonite protects your entire IT footprint with the right level of protection for each workload, ensuring lower costs and dependable solutions with DoubleTake and Evault.
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs oft...
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists will examine how DevOps helps to meet th...
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value S...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deli...
We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that Infranics will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Since 2000, Infranics has developed SysMaster Suite, which is required for the stable and efficient management of ICT infrastructure. The ICT management solution developed and provided by Infranics continues to add intelligence to the ICT infrastructure through the IMC (Infra Management Cycle) based on mathemat...
SYS-CON Events announced today that HTBase will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. HTBase (Gartner 2016 Cool Vendor) delivers a Composable IT infrastructure solution architected for agility and increased efficiency. It turns compute, storage, and fabric into fluid pools of resources that are easily composed and re-composed to meet each application’s needs. With HTBase, companies can quickly prov...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 21st International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. @ThingsExpo Silicon Valley Call for Papers is now open.
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...