Future Data Center Architectures

Check out new video uploaded by Darren at October 29, 2019 at 04:06AM. In this episode, the history of data center architecture and application development is reviewed and the trends of application development shaping the data center of the future. Find out how containers, serverless, and data mesh architectures are being leveraged to decrease deployment times and increase reliability.

Billy Jean Greek style

Check out new video uploaded by Darren at October 17, 2019 at 12:50PM. Paige and Darren were out walking the streets of Athens near Parliament Square and found this incredible performer. Check his incredible rendition of Billy Jean.

2019 09 09 14 32 26 1

Check out new video uploaded by Darren at September 19, 2019 at 05:38PM. This week Paige continues to teach Darren what she learned at "Education Week". This episode we tackle managing problems, not solving problem, managing them. Some problems, like putting socks in the hamper, take a long to solve, but all problems can be managed.

VideoCast 1:6 - Elastic Search and DCPMM

Check out new video uploaded by Darren at September 13, 2019 at 08:18AM. Intel's new Persistent Memory technology has three modes of operation. One as an extension of your current memory. Imagine extending your server with 9TBytes of Memory. The second is called AppDirect mode where you can use the Persistent Memory as a persistent segment of memory or as a high-speed SSD. The third mode is called mix mode. In this mode, a percentage of the persistent memory is used for AppDirect and the other to extend your standard DDR4 Memory. When exploring this new technology, I realized that I could take the persistent memory and use it as a high-speed SSD. If I did that could I increase the throughput of my ElasticSearch Server? So I set up a test suite to try this out.

ElasticSearch with Intel Optane DCPMM

Check out new video uploaded by Darren at September 10, 2019 at 04:59PM. Want to improve you ElasticSearch performance. Check out using Intel's Optane DCPMM as a fast filesystem and watch Elastic fly.

Podcast 1:5 - Information Management Maturity Model

Check out new video uploaded by Darren at August 29, 2019 at 03:42PM. Developing a Data Strategy can be difficult, especially if you don’t know where your current organization is and where it wants to go. The Information Management Maturity Model helps CDOs and CIOs find out where they currently are in their Information Management journey and their trajectory. This map helps guide organizations as they continuously improve and progress to the ultimate data organization that allows them to derive maximum business value from their data.

Episode 1:4 - History of Data Architecture

Check out new video uploaded by Darren at August 7, 2019 at 11:07PM. Organizations are looking to their vast data stores for nuggets of information that give them a leg up on their competition. “Big Data Analytics” and Artificial Intelligence are the technologies promising to find those gold nuggets. Mining data is accomplished through a “Distributed Data Lake Architecture” that enables cleansing, linking, and analytics of varied distributed data sources. Rise of the Stack Developer Episode 4 August 8, 2019 ★ Episode details: https://ift.tt/2GWgpSp ★ Additional episodes: https://ift.tt/2K6XtBj

Robotic Process Automation in Multi-Hybrid Clouds

Check out new video uploaded by Darren at August 2, 2019 at 12:08PM. One of the growing areas to help with Legacy Integration and automation of integration is the use of automation tools and frameworks. Over the last 3 years, a significant emphasis on the automation of workflows with legacy and new cloud-aware applications for information workers has emerged. These tools sets are called Robotic Process Automation (RPA) tools.

Podcast 1:3 - Decreasing Ingestion Congestion with Optane DCPMM

Check out new video uploaded by Darren at July 31, 2019 at 02:21PM. Big Data analytics needs data to be valuable. Collecting data from different machines, IoT devices, or sensors is the first step to being able to derive value from data. Ingesting data with Kafka is a common method used to collect this data. Find out how using Intel's Optane DC Persistent Memory to decrease ingestion congestion and increase total thruput of your ingestion solution. Rise of the Stack Developer Episode 3 July 31, 2019 ★ Episode details: https://ift.tt/2KkZQR5 ★ Additional episodes: https://ift.tt/2K6XtBj

Podcast 1:3 - Decreasing Ingestion Congestion with Optane DCPMM

Big Data analytics needs data to be valuable. Collecting data from different machines, IoT devices, or sensors is the first step to being able to derive value from data. Ingesting data with Kafka is a common method used to collect this data. Find out how using Intel's Optane DC Persistent Memory to decrease ingestion congestion and increase total thruput of your ingestion solution.

Kafka in real examples

• Ingestion is the first step in getting data into your data-lake or data warehouse
• Kafka is basically a highly available distributed PubSubHub.
• Data from a producer is published on Kafka Topics which consumers subscribe to. Topics give the ability segment the data for ingestion.
• Topics can be spread over a set of servers on different physical devices to increase reliability and thruput.
• Performance Best Practices
• Buffer Size of the Producers should be a multiple of the message size
• Batch Size can be changed based on the message size for optimal performance
• * Spread Logs for partitions across multiple drives or on fast drives.
• Example Configuration (LinkedIn)
• 13 million messages per second, (2.75 GB/s) 
• 1100 Kafka brokers organized into more than 60 clusters.
• Automotive Space
• One Customer has 100 Million Cars - 240KB/min/car
• 1.6 Million Messages/sec
• 800 GB/s
• Approximate size of the installation
• 4400 Brokers, over 240 Clusters.

Optane DC Persistent Memory

• Ability to use Optane technology in a DDR4 DIMM form factor.
• 128GB, 256 GB, 512GB PMMs are available. Up to 3 TB per socket
• Two modes of operation: App Direct Mode, and Memory Mode.
• Memory Mode gives the ability to have cheaper memory than typical DDR4 prices at a fraction of the cost. 
• App Direct Mode means you can write a program to write directly to memory and it is persistent. Survives over reboots or power loss.
• App Direct Mode can also be used to create ultra-fast filesystems with memory drives.
• DCPMM uses DDR4 memory  and DCPMM in a mixed mode. Example a 16G DIMM paired with a 128G PMM. or a 64G DIMM Paired with a 512GB PMM.
• Memory modes can be controlled in the Bios of from the linux kernel. 
• ipmctl - utility for configuring and managing Intel Optane DC persistent memory modules (PMM).
• ndctl – utility for managing (non-volatile memory device) sub-system in the Linux kernel

Improving Ingestion using Persistent Memory

• Use Larger Memory Footprint for more kafa servers on the same machine with larger Heap Space
• Change Kafka to write directly to Persistent Memory
• Create a Persistent Memory Filesystem and point kafka logs to the new filesystem

Testing Results

• Isolate performance variability by limiting the testing to one broker on the same machine as the producer.
• Remove network variability and bottleneck of the network.
• Decrease inter-broker communication and replica bottlenecks
• Only producer is run to find the maximum that can be ingested.
• Only Consumers are run to find the maximum that can be egressed.
• Mixed Producer and Consumer are run to find passthru rates.
• First approach. 50% persistent memory in App Direct Mode
• 3x performance over Sata Drive mounted log files
• 2x performance over Optance NVMe drives
• Second approach. 100% persistent memory in App Direct Mode
• 10x performance over Sata Drive mounted log files.
• approximately ~2 Giga Bytes per second. over 150 MB/sec for SATA drive
• Additional testing has been performed with Cluster to increase total thruput and we found we were limited not by the drive speed which is normally the case, but by the network speed. We were limited to 10 G bit network.

Podcast 1:2 - Legacy integration with Robotic Process Automation

Check out new video uploaded by Darren at July 27, 2019 at 04:30PM. One of the growing areas to help with Legacy Integration and automation of integration is the use of automation tools and frameworks. Over the last 3 years, a significant emphasis on the automation of workflows with legacy and new cloud-aware applications for information workers has emerged. These tools sets are called Robotic Process Automation (RPA) tools. Rise of the Stack Developer Episode 2 July 27, 2019 ★ Episode details: https://ift.tt/32XIUZq ★ Additional episodes: https://ift.tt/2K6XtBj

Rise of the Stack Developer History

Check out new video uploaded by Darren at July 27, 2019 at 04:29PM. System Administration, Configuration Management, Build Engineer, and DevOps have many of the same responsibilities but over the years the names have changed. Listen as Darren Pulsipher gives a brief history of his journey through the software and product development over the last four decades and how so much has changed and much has remained the same. Rise of the Stack Developer Episode 1 May 6, 2019 ★ Episode details: https://ift.tt/32SlAfH ★ Additional episodes: https://ift.tt/2K6XtBj

Decreasing Kakfa Ingestion bottlenecks using Intel Optane DC Presistent Memory

Check out new video uploaded by Darren at May 22, 2019 at 12:48AM. This video shows how to use Intel's Optane DC Persistent Memory to create a persistent memory mapped drive and use it as a filesystem to store Kafka log files. Look at the 5-10x performance improvement you get with this technique.

Decreasing Kafka Ingestion Congestion with Intel Optane DC Persistent Memory

Check out new video uploaded by Darren at February 19, 2019 at 04:34PM. This video explains one of the primary challenges of autonomous driving is how to effectively receive, manage, and mine vast quantities of data to help ensure safe driving outcomes. Learn how Intel and Red Hat have been working jointly in the areas of storage and memory to deliver a strong foundation—based on Red Hat* Enterprise Linux 7.6 supporting Intel® Optane™ DC persistent memory and Intel® Optane™ DC SSDs—to meet these challenges.