SD Times news digest: ArrangoML Pipeline Cloud, Hyperledger Fabric 2.0, and Pandas 1.0

ArangoDB has announced a new machine learning offering. The ArangoML Pipeline Cloud is a fully-hosted, fully-managed common metadata layer for production-grade data science and ML platforms that runs on ArangoDB Oasis.

“ArangoML Pipeline Cloud meets the needs of both data scientists, who are concerned with the quality of the data, feature training, and model results, as well as DevOps, who need to manage which datasets and deployments are in use, their performance, and how they are being deployed,” ArangoDB wrote in a post.

ArangoDB can easily accommodate and unite unstructured, highly-interlinked data, such as inference and model descriptions and allow relationships between them to be stored as a graph that can be managed by a DevOps engineer and be used by a data scientist at the same time, the company explained.

Hyperledger Fabric 2.0 released

Hyperledger Fabric 2.0’s improvements were focused on new ways to manage the chaincode (or smart contract) life cycle to maximize flexibility, remove bottlenecks and create more options to distribute governance.

Security and data privacy were also big development priorities, so many features and enhancements are geared towards delivering more granular control of private channels and minimizing vulnerabilities, according to the team.

Key Hyperledger Fabric 2.0 features include decentralized governance for smart contracts, new chaincode application patterns for collaboration and consensus, an external chaincode launcher, private data enhancements, state database cache for improved performance on CouchDB, and Alpine-based docker images.

Pandas 1.0 announced

Pandas is a open-source library for easy-to-use Python data structures and data analysis. The new release removed a lot of functionality that was deprecated in previous releases and includes enhancements and new features.

Enhancements include the ability to use Numba in ‘rolling.apply’ and ‘expanding.apply,’ the ability to define custom windows for rolling operations, and creating markdown tables.

Experimental new features include NA scalar to denote missing values, a dedicated string data type, and Boolean data types with missing values support.

The full details are available here.

Stoplight closes $6 million series A funding round

API Design Management company Stoplight announced $6 million in a series A funding round, bringing the company’s total funding to $10 million.

Stoplight plans to utilize the funding to augment engineering efforts, accelerate its product roadmap and cover more customer use cases.

“API-first design is becoming a fundamental component of enterprise engineering, and we have full confidence that Stoplight will only deepen their category leadership in the next several years,” said Mike Smerklo, co-founder and managing director of Next Coast Ventures.

Related:

Hyperledger blockchain project releases Hyperledger Fabric 2.0

Open-source collaborative blockchain project Hyperledger has announced the launch of Hyperledger Fabric 2.0.

Hyperledger Fabric is a distributed ledger framework that has been under development by the Hyperledger community since 2016 and the v1.0 was released in July 2017. It has been widely deployed in PoCs and production networks and adopted by major cloud service providers including Alibaba, AWS, Azure, Baidu, Google, Huawei, IBM, Oracle, SAP, and Tencent. It is the first Hyperledger project to hit the 2.0 milestone.

“Fabric 2.0 is a new generation framework developed by and for the enterprises that are building distributed ledger capabilities into the core of their businesses. This new release reflects both the development and deployment experience of the Fabric community and confirms the arrival of the production era for enterprise blockchain,” said Brian Behlendorf, Executive Director, Hyperledger.

In a press release, Hyperledger said that this major release includes various features and enhancements required for enhanced efficiency and security of production deployments. The key focus was on exploring new ways to manage the chaincode (or smart contract) lifecycle to maximize flexibility, remove bottlenecks and create more options to distribute governance, along with security and data privacy.

According to the details, key Hyperledger Fabric 2.0 features include decentralized governance for smart contracts, new chaincode application patterns for collaboration and consensus, external chaincode launcher, private data enhancements, state database cache for improved performance on CouchDB, and Alpine-based docker images.

“The release of Hyperledger Fabric 2.0 is an important step forward in the on-going evolution of DLT, and was developed based on feedback from real-world use, including improved chaincode management capabilities and performance enhancements,” stated Rob Palatnick, Managing Director and Global Head of Technology Research and Innovation at The Depository Trust & Clearing Corporation (DTCC) and Governing Board Chair at Hyperledger.

Detect languageAfrikaansAlbanianAmharicArabicArmenianAzerbaijaniBasqueBelarusianBengaliBosnianBulgarianCatalanCebuanoChichewaChinese (Simplified)Chinese (Traditional)CorsicanCroatianCzechDanishDutchEnglishEsperantoEstonianFilipinoFinnishFrenchFrisianGalicianGeorgianGermanGreekGujaratiHaitian CreoleHausaHawaiianHebrewHindiHmongHungarianIcelandicIgboIndonesianIrishItalianJapaneseJavaneseKannadaKazakhKhmerKoreanKurdishKyrgyzLaoLatinLatvianLithuanianLuxembourgishMacedonianMalagasyMalayMalayalamMalteseMaoriMarathiMongolianMyanmar (Burmese)NepaliNorwegianPashtoPersianPolishPortuguesePunjabiRomanianRussianSamoanScots GaelicSerbianSesothoShonaSindhiSinhalaSlovakSlovenianSomaliSpanishSundaneseSwahiliSwedishTajikTamilTeluguThaiTurkishUkrainianUrduUzbekVietnameseWelshXhosaYiddishYorubaZulu
AfrikaansAlbanianAmharicArabicArmenianAzerbaijaniBasqueBelarusianBengaliBosnianBulgarianCatalanCebuanoChichewaChinese (Simplified)Chinese (Traditional)CorsicanCroatianCzechDanishDutchEnglishEsperantoEstonianFilipinoFinnishFrenchFrisianGalicianGeorgianGermanGreekGujaratiHaitian CreoleHausaHawaiianHebrewHindiHmongHungarianIcelandicIgboIndonesianIrishItalianJapaneseJavaneseKannadaKazakhKhmerKoreanKurdishKyrgyzLaoLatinLatvianLithuanianLuxembourgishMacedonianMalagasyMalayMalayalamMalteseMaoriMarathiMongolianMyanmar (Burmese)NepaliNorwegianPashtoPersianPolishPortuguesePunjabiRomanianRussianSamoanScots GaelicSerbianSesothoShonaSindhiSinhalaSlovakSlovenianSomaliSpanishSundaneseSwahiliSwedishTajikTamilTeluguThaiTurkishUkrainianUrduUzbekVietnameseWelshXhosaYiddishYorubaZulu

Use a PostgreSQL database as a Hyperledger Fabric wallet using Fabric Node SDK

Summary

Hyperledger Fabric uses appropriate enrollment certificates while interacting with the blockchain network. The Hyperledger Fabric SDK for Node.js provides APIs to interact with a Hyperledger Fabric blockchain. The Fabric Node SDK provides a default file system wallet for storing Fabric certificates. The file system wallet stores user certificate in folders. This approach does not provide the required security or flexibility, and it also affects scalability. Explore this pattern further to understand how to use a PostgreSQL database as a Fabric wallet.

Description

Hyperledger Fabric is one of the blockchain projects within Hyperledger. It is private and permissioned; security on Hyperledger Fabric is enforced with digital signatures. All requests made to Hyperledger Fabric must be signed by users with appropriate enrollment certificates. Once a user is enrolled, Node.js saves the certificates in the wallet for future use. A wallet contains a set of user identities. An application run by a user selects one of these identities when it connects to a channel.

There are four types of wallets: file system, in-memory, hardware security module, and CouchDB. The Hyperledger Fabric SDK for Node.js provides a default file-system wallet for storing Fabric certificates, storing users’ certificates in folders. The Hyperledger Fabric SDK also provides a way to configure a wallet in CouchDB.

But what if a user wants to use the PostgreSQL database instead of CouchDB? There is no direct provision for storing enrollment certificates to PostgreSQL. The PostgreSQL database supports SQL and NoSQL data storage and has strong community support. This pattern demonstrates a methodology for using the PostgreSQL database as a wallet within the Hyperledger Fabric SDK for Node.js. We use IBM Blockchain Platform to set up the network and PostgreSQL configured as an IBM Cloud service or containerized using Kubernetes.

Flow

flow

  1. Set up a Hyperledger Fabric network using IBM Blockchain Platform.
  2. Configure and deploy a containerized PostgreSQL database using Kubernetes.
  3. Deploy the client application using the Hyperledger Fabric SDK for Node.js, where users can communicate with the blockchain network.
  4. To communicate with the blockchain network, users need to register and enroll with the network which will generate the enrollment certificates and store them in a PostgreSQL database. These certificates will be used for further communication with the network.

Instructions

Ready to give it a try? Check out the README for detailed instructions.

Related:

Explore the querying capability of Hyperledger Fabric 1.4

Summary

In this developer code pattern, we will take a look at how to query the world state of a peer within Hyperledger Fabric. The world state, implemented as a database, contains all of the network history (e.g., transactions issued). Querying the world state is useful for seeing the current state of the assets in the network. We will use the commercial paper use case from the Hyperledger Fabric 1.4 documentation. You’ll go through the process of creating indexes for the CouchDB world state database, update the smart contract to include the logic to query the world state using the newly created indexes, simulate 100 transactions to populate the world state with assets, then run a few queries utilizing the Node.js SDK and view the returned results.

Description

Before we jump into showing how to query the world state, we first need to understand what database indexes are and how they can help us with queries.

In order to understand indexes, let’s take a look at what happens when you query the world state. Say, for example, you want to find all assets owned by the user “Bob.” The database will search through each JSON document in the database one by one and return all documents that match user = bob. This might not seem like a big deal, but consider if you have millions of documents in your database. These queries might take a while to return the results since the database needs to go through each and every document. With indexes, you create a reference that contains all the values of a specific field and which document contains that value. What this means is that instead of searching every document, the database can just search the index for occurrences of the user “bob” and return the documents that are referenced. It’s important to note that every time a document is added to the database, the index needs to be updated. Normally in CouchDB, this is done when a query is received, but in Hyperledger Fabric, the indexes are updated every time a new block is committed, which allows for faster querying. This is a process known as index warming.

Flow

flow

  1. The developer creates the query indexes.

  2. The developer adds query logic to the smart contract.

  3. The IBM Blockchain extension for VSCode is then used to package, install, and instantiate the smart contract and indexes on the local Hyperledger Fabric network.

  4. We simulate 100 transactions on the network to populate the world state with data.

  5. The Node.js client application uses the Hyperledger Fabric SDK to send the query to the endorsing nodes to evaluate the query transaction.

  6. The query results are returned.

Instructions

Ready to get started? Check out the README for detailed instructions.

Related:

IBM Answers Questions About Hyperledger Fabric Blockchain Performance and Scale

IBM Answers Questions About Hyperledger Fabric Blockchain Performance and Scale

In a recent blog post, IBM aims at answering the questions and doubts that several individuals in the cryptocurrency market had about the Linux Foundation’s Hyperledger Fabric performance and scale. According to the company, there has been a lot of confusion and FUD related to this issue.

According to IBM Hyperledger Fabric does not have problems with scaling. However, it might not offer the results expected for specific applications. Thus, the company explains that the results will have an impact depending on the kind of use that firms give to it.

Christopher Ferris, IBM Distinguished Engineer, and CTO Open Technology in the IBM Digital Business Group, started to write a series of articles to provide best practices guidance and the latest information related to the Hyperledger Fabric.

He starts the post by explaining that when considering the performance of any blockchain framework there are several factors that can affect the framework itself. The first thing that Ferris mentions is that there is an application client. The programming language in which this client was written will have an impact on the choice of Hyperledger Fabric software developer kit (SDK). Furthermore, the client itself can have some issues since there is no software defect-free, according to Ferris.

The second thing that he shares is that there is the Hyperledger Fabric peer (endorser/committer) and choice of ledger database. The two ledger databases are LevelDB and CouchDB.

Thirdly, there are several chaincode implementation choices such as Go, JavaScript, Java and others. Fourth, the ordering services node can also affect the performance of the network. The fifth thing Ferris acknowledged is that there is the distribution of work within a blockchain network based on how the creators architected use of channels and/or private transactions to deliver privacy.

Finally, he says that there is a physical and/or virtual infrastructure that all of the services run on that can severely affect performance.

Users and individuals have been asking several questions regarding what can have an effect on the performance of the Hyperledger Fabric. Some of the things asked by users were the number of endorsing peers, the number of endorsements, the number of organizations, the complexity of chaincode, the size of transactions, the memory allocation, the network speed and many others.

Although Ferris will not be trying to address all these questions in just an article, he will be trying to provide some answers to these and other questions.

The first question that he will answer is ‘Does the number of endorsers effect performance?.’ About it, he says that yes, it will affect the performance depending on the network composition.

In order to provide a full answer, Ferris gives a very interesting example with some assumptions.

“Say you have a network that has two organizations with two peers each for resilience. These peers are each joined to a channel comprised of the two organizations. […] Let’s say that the endorsement policy requires a single endorsement from either organization. Let’s further start with only one of each organization’s peers configured as an endorser.”

He went on saying that the transaction load generator will be randomly selecting one of these endorsers for each transaction proposal. Ferris says that for this experiment, the network will be running Hyperledger Fabric 1.3.0 in a single Kuebrnetes cluster running on the IB Container Service.

The results were the following:

Ferris shows that it is possible to scale throughput by load balancing endorsement across a pool of endorsers.

He noted that there will be a moment in which transaction latency becomes untenable due to the fact that peers become saturated and consume all the CPU available. According to Ferris, it is possible to scale beyond the number of TPS shown in the results, if they decide to tolerate more latency.

He has also talked about another myth that spread that says that Hyperledger Fabric limits the number of peer nodes in a network. He says that this is very far from being the truth.

In the future, Christopher Ferris will be addressing several other issues, some of those were mentioned in the article. This would help interested parties to understand how Hyperledger Fabric works and how it scales properly in case it is needed.

7022979: How to restore Crowbar backup on the admin node

Following steps will clean and restore backup without need to reinstall the admin node
0. Create a snapshot prior cleaning up (optional for btrfs)
snapper create -d "cleaned admin node"

1. Check admin node uses localhost and ip of admin server as dns servers.
Can be restored from cache if needed:

cp -a /var/lib/crowbar/cache/etc/resolv.conf /etc/resolv.conf

DNS forwarders can be added to
/etc/bind/named.conf

Example resolv.conf:

cat >/etc/resolv.conf <<EOF

search examplecloud.suse.com

nameserver 127.0.0.1

nameserver 192.168.124.10

EOF

2. Stop all cloud services
for service in crowbar crowbar-jobs crowbar-init
tftp chef-{server,solr,expander,client} couchdb
apache2 named dhcpd xinetd rabbitmq-server postgresql dnsmasq;
do
systemctl stop $service;
done
2.1 Stop unstopped processes (if they exists)

killall epmd # part of rabbitmq

killall looper_chef_client.sh

3. Deinstall cloud packages

zypper rm

`rpm -qa|grep -e crowbar -e chef -e cloud -e apache2`

couchdb createrepo erlang rabbitmq-server sleshammer yum-common

bind bind-chrootenv dhcp-server tftp

4. Delete content in cloud directrories

rm -rf /opt/dell/*

/etc/{bind,chef,crowbar,crowbar.install.key,dhcp3,xinetd.d/tftp}

/etc/sysconfig/{dhcpd,named/*,rabbitmq-server}

/var/lib/{chef,couchdb,crowbar,dhcp,rabbitmq}

/var/run/{chef,crowbar,named/*,rabbitmq}

/var/log/{apache2,chef,couchdb,crowbar,nodes,rabbitmq}

/var/cache/chef

/var/chef

/var/lib/pgsql/*

/srv/tftpboot/{discovery/pxelinux.cfg/*,nodes,validation.pem}

(Note: In case of non btrfs root partition some directrories needs to be recreated)
5. Verify subprocesses are also down after deinstallation (usually not needed anymore)

killall epmd

############ #clean state #############
6. Create snapshot of cleaned admin node (optional for btrfs)

snapper create -d "cleaned admin node"

7. Ensure admin node is resolvable, nslookup and ping works

#e.g. start dnsmasq and add ip/name to /etc/hosts as needed

systemctl start dnsmasq.service
8. Install required patterns
zypper in -t pattern base cloud_admin


9. apply PTF fix for hostname check against /etc/hostname instead "hostname -f" if still needed

rpm -Fhv /ptf/*.rpm

10. Prepare backup in place

mkdir restore

cd restore

tar -xvf ../backup-restore.tar.gz

10.1 Make sure admin node is network.json compliant - the admin ip matches
first ip of an interface ( usually eth0 )
cp crowbar/configs/crowbar/network.json /etc/crowbar/network.json


10.2 Copy/Add hostnames to "/etc/hosts" from backup if not sufficient

cp crowbar/configs/hostname /etc/hostname

10.3 If using remote SMT add ip/name to the backup tar (not needed if reachable

via separate bastion network)

echo "10.10.10.10 smt-server.suseexample.com smt-server">> crowbar/configs/hosts

echo "10.10.10.0 192.170.124.1" >> /etc/sysconfig/network/routes



10.4 Create new tar-archive with changes

tar -czvf restore.tar.gz crowbar/ knife/ meta.yml

11 Start crowbar-init

systemctl start crowbar-init

12 Create empty crowbar database

crowbarctl database create

(To check sanity checks what causes issues access via browser:

http://localhost:3000/sanity)

######## #restore ########
13 Create snapshot prior restore (optional for btrfs)

snapper create -d "clean admin node prior uploading/restore backup"



14 Upload restore tarball
crowbarctl backup upload restore.tar.gz --debug --anonymous

(Note:if it still does not work check to use fqdn in /etc/hostnames or get PTF installed
15 Restore backup

crowbarctl backup list

crowbarctl backup restore <name-from-list>

16 monitor progress

tail -f /var/log/apache2/*log /var/log/crowbar/*

(or access installer within browser

17 finally check services are up, if not start them:

for service in crowbar crowbar-jobs

tftp chef-{server,solr,expander,client} couchdb
apache2 named dhcpd xinetd rabbitmq-server postgresql dnsmasq;
do
systemctl status $service;
done

Related: