DD Boost for Enterprise Applications: DDBMEXPTOOL doesn’t delete expired data when -N is used

Article Number: 503010 Article Version: 3 Article Type: Break Fix



DD Boost for Enterprise Applications,Microsoft App Agent

The backups are not all deleted even though they are expired,

when running DDBPEXPTOOL or the SQl job returns successful however not all backups are not deleted reuslting in DD not cleaned up

USE [master]

GO

DECLARE @returnCode int

EXEC @returnCode = dbo.emc_run_delete ‘-e “1 days ago” -k -n mssql -a “DDBOOST_USER = DDBOOST_USER” -a “DEVICE_HOST = DD_HOSTNAME ” -a “DEVICE_PATH = /STorage_UNIT” -a “CLIENT = CLient_name” -N “BAckup_Name “‘

ddbmexptool -k [-v -D 9 -b <start_save_time> -e <end_save_time> -N <save_set_name>] {-z <config_file> | -a “parameter=value”} -n <application>


when using the following command

ddbmexptool -k [-v -D 9 -b <start_save_time> -e <end_save_time> {-z <config_file> | -a “parameter=value”} -n <application>

or the T-SQL wihtout -N

USE [master]

GO

DECLARE @returnCode int

EXEC @returnCode = dbo.emc_run_delete ‘-e “1 days ago” -k -n mssql -a “DDBOOST_USER = DDBOOST_USER” -a “DEVICE_HOST = DD_HOSTNAME ” -a “DEVICE_PATH = /STorage_UNIT” -a “CLIENT = CLient_name” ‘


The expired data are deleted as expected.



This is caused by the delete command filtering on the name of the backup and only deleting backup’s which have the name -N “Backup_Name”‘

-N specifies an exact match, not a partial match. So if you are specifying -N “Backup_Name” it will ignore anything with a database name appended onto the end.

in the list command returning all of the backup’s made because didn’t add a saveset name filter.

USE [master]

GO

DECLARE @returnCode int

EXEC @returnCode = dbo.emc_run_delete ‘-e “1 days ago” -k -n mssql -a “DDBOOST_USER = DDBOOST_USER” -a “DEVICE_HOST = DD_HOSTNAME ” -a “DEVICE_PATH = /STorage_UNIT” -a “CLIENT = CLient_name”‘



Run the expiry tool or the Maintenance T-SQL Transcript without specifying the -N Parameter

USE [master]

GO

DECLARE @returnCode int

EXEC @returnCode = dbo.emc_run_delete ‘-e “1 days ago” -k -n mssql -a “DDBOOST_USER = DDBOOST_USER” -a “DEVICE_HOST = DD_HOSTNAME ” -a “DEVICE_PATH = /STORAGE_UNIT” -a “CLIENT = Client_name” ‘

ddbmexptool -k [-v -D 9 -b <start_save_time> -e <end_save_time> {-z <config_file> | -a “parameter=value”} -n <application>

Related:

  • No Related Posts

Hyperledger Fabric 1.4 released with long term support

Today Hyperledger part of the Linux Foundation announced the release of the latest version of enterprise blockchain Hyperledger Fabric 1.4 LTS. The LTS stands for long term support and is Fabric’s first such release. Commonly with operating systems and major software distributions particular versions are badged as LTS which means security patches are released for that version even after new versions with added features become available.

This is important in a production environment when upgrading to new versions involves a lot of work to adapt code. Particularly if developers don’t necessarily want to take advantage of new features quite yet. Hence Hyperledger is committing to release patches for Hyperledger Fabric 1.4 until 10 January 2020. By LTS standards, a year is quite short, but blockchain is a fast moving sector. Linux versions are typically supported for five years and major software frameworks for roughly two years.

One area that Hyperledger has received a little flak is the serviceability for a production environment. Most servers store logs of key activities and the ability to interrogate these logs is critical for debugging issues. Hence Fabric 1.4 includes enhanced logging, health checks and operational metrics. The team referred to the serviceability as a “giant leap forward” in the announcement.

Previous versions of Fabric included an SDK and chaincode for Node.js the popular server-side javascript scripting framework. The programming model for both of these has a major upgrade so that developers can focus more on the application logic rather than needing to delve deep into the nitty-gritty.

Additionally, 1.4 features privacy enhancements. Latecomers added to private data collections can now retrieve private data for historical transactions which they’re entitled to see. And client access control is now automatically enforced within chaincode by a simple configuration without the need to write specific code.

The race is on

Hyperledger version 2.0 is expected in the Spring. It includes major features such as enhanced support for tokens and atomic swaps. Plus there’s an additional consensus mechanism – Raft – being added to existing consensus options Solo and Kafka.


Image Copyright: hyperledger

Related:

  • No Related Posts

Virus Definition not Updating; Fix Fails on Endpoint

I need a solution

 I have 3 machines out of 50+ that are showing as having out-of-date virus definitions in the cloud console. Two of the machines are Windows 10 while the third is Windows 7. 

I go to each machine and click on the “Fix” button. The fix does not work. 

I can quick and full scan, run live update, etc.. In fact I can do everythign except get the virus update definitions to update and the the out-of-date virus definition error to go away. 

Short of re-installing the Endpoint client, do I have any other options?

0

Related:

  • No Related Posts

Microsoft Windows Security Updates January 2019 overview

Welcome to the first Microsoft Windows Patch Day overview of 2019. Microsoft released security updates for all supported client and server versions of the Windows operating system and other company products such as Microsoft Office on January 8, 2019.

We publish a monthly overview shortly after Microsoft’s release on the second Tuesday of each month. The overview lists all released security updates with links to Microsoft Support articles, known issues, downloads, and other Patch Tuesday related information.

You can check out the December 2018 Patch Day overview here.

Note: As always, we recommend to back up the system before you install updates for Windows or any other program.

Microsoft Windows Security Updates January 2019

Click on the following link to download an Excel spreadsheet that includes data about all released security updates for Microsoft Windows versions and other Microsoft products. Just click on the following link to start the download: security-updates-microsoft-january-2019-windows.zip

Executive Summary

  • Microsoft released security updates for all client and server versions of Windows.
  • No critical vulnerabilities in Windows 8.1 and 7.
  • Microsoft released security updates for Microsoft Edge, Internet Explorer, Adobe Flash Player, .NET Framework, Microsoft Office, Microsoft Exchange Server, and Microsoft Visual Studio
  • Windows 10 version 1809 is in active distribution. Check out our guide on delaying feature updates for Windows 10 to avoid the installation.
  • The Update Catalog lists 187 updates for January 2019.

Operating System Distribution

  • Windows 7: 15 vulnerabilities of which 15 are rated important.
  • Windows 8.1: 18 vulnerabilities of which 18 are rated important.
  • Windows 10 version 1607: 23 vulnerabilities of which 1 is critical and 22 are important
  • Windows 10 version 1703: 24 vulnerabilities of which 1 is critical and 23 are important
  • Windows 10 version 1709: 24 vulnerabilities of which 1 is critical and 23 are important
  • Windows 10 version 1803: 26 vulnerabilities of which 3 are critical and 23 are important
  • Windows 10 version 1809: 25 vulnerabilities of which 2 are critical and 23 are important

Windows Server products

  • Windows Server 2008 R2: 15 vulnerabilities of which 15 are important.
  • Windows Server 2012 R2: 18 vulnerabilities of which 18 are important.
  • Windows Server 2016: 23 vulnerabilities of which 1 is critical and 22 are important.
  • Windows Server 2019: 25 vulnerabilities of which 2 are critical and 23 are important.

Other Microsoft Products

Windows Security Updates

All Windows versions:

Starting with the January 2019 security updates, PowerShell remote endpoints cannot be configured anymore to work with non-administrator accounts.

Attempts to use non-admin accounts throws the following error after installation of the updates:

“New-PSSession: [computerName] Connecting to remote server localhost failed with the following error message: The WSMan service could not launch a host process to process the given request. Make sure the WSMan provider host server and proxy are properly registered. For more information, see the about_Remote_Troubleshooting Help topic.”

Windows 10 version 1809

KB4480116

Security updates to Microsoft Edge, Internet Explorer, Windows App Platform and Frameworks, Windows MSXML, Windows Kernel, Windows Storage and Filesystems, Windows Wireless Networking, Microsoft JET Database Engine, Windows Linux, Windows Virtualization, and the Microsoft Scripting Engine.

Windows 10 version 1803

Fixes a highly exploitable issue in Windows 10 version 1803; recommended to patch as early as possible. See Zero Day Initiative and Microsoft’s guidance on the vulnerability.

KB4480966

  • Security updates to Internet Explorer, Microsoft Edge, Windows App Platform and Frameworks, the Microsoft Scripting Engine, Windows Kernel, Windows Storage and Filesystems, Windows Virtualization, Windows Linux, Windows MSXML, and the Microsoft JET Database Engine .

Windows 10 version 1709

KB4480978

  • Fixes an issue with esentutl /p which caused the repair to result in a “mostly empty database” which is corrupt and cannot be mounted.
  • Security updates to Internet Explorer, Microsoft Edge, Windows App Platform and Frameworks, the Microsoft Scripting Engine, Windows MSXML, Windows Kernel, Windows Storage and Filesystems, Windows Virtualization, Windows Linux, and the Microsoft JET Database Engine.

Windows 10 version 1703

KB4480973

  • Security updates to Internet Explorer, Microsoft Edge, Windows App Platform and Frameworks, the Microsoft Scripting Engine, Windows Authentication, Windows MSXML, Windows Kernel, Windows Storage and Filesystems, Windows Virtualization, Windows Linux, and Microsoft JET Database Engine

Windows 10 version 1607

KB4480961

  • Security updates to Internet Explorer, Windows App Platform and Frameworks, the Microsoft Scripting Engine, Windows Kernel, Windows Hyper-V, Windows MSXML, and the Microsoft JET Database Engine.

Windows 8.1 and Windows Server 2012 R2

KB4480963 Monthly Rollup

  • Protection against Speculative Story Bypass CVE-2018-3639 for AMD-based computers
  • Security updates to Windows App Platform and Frameworks, Windows MSXML, Windows Kernel, Windows Storage and Filesystems, Windows Wireless Networking, and the Microsoft JET Database Engine.

KB4480964 Security-only

  • Same as Monthly Rollup

Windows 7 SP1 and Windows Server 2008 R2 SP1

Note: The updates may introduce issues with network shares.

KB4480970 Monthly Rollup

  • Protection against Speculative Story Bypass CVE-2018-3639 for AMD-based computers
  • Security updates to Windows Kernel, Windows Storage and Filesystems, Windows Wireless Networking, and the Microsoft JET Database Engine.

KB4480960 — Security-only

  • Same as Monthly Rollup

Other security updates

KB4483235 — Windows 10 version 1809 and Windows Server 2019 — Security update for Internet Explorer

KB4483234 — Windows 10 version 1803 — Security update for Internet Explorer

KB4483232 — Windows 10 version 1709 — Security update for Internet Explorer

KB4483230 — Windows 10 version 1703 — Security update for Internet Explorer

KB4483229 — Windows 10 version 1607 and Windows Server 2016 — Security update for Internet Explorer

KB4483187 — Cumulative security update for Internet Explorer: December 19, 2018 — fixes a remote code execution vulnerability.

KB4480059 — Security and Quality Rollup for .NET Framework 4.5.2 for Windows Embedded Standard 7, Windows 7, Windows Server 2008 R2, and Windows Server 2008

KB4480051 — Security and Quality Rollup for .NET Framework 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows Embedded 8 Standard and Windows Server 2012

KB4480054 — Security and Quality Rollup for .NET Framework 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows 8.1 and Windows Server 2012 R2

KB4480055 — Security and Quality Rollup for .NET Framework 4.6 for Windows Embedded Standard 7, Windows 7, Windows Server 2008 R2, and Windows Server 2008

KB4480057 — Security and Quality Rollup for .NET Framework 4.5.2 for Windows 8.1 and Windows Server 2012 R2

KB4480058 — Security and Quality Rollup for .NET Framework 4.5.2 for Windows Embedded 8 Standard and Windows Server 2012

KB4480061 — Security and Quality Rollup for .NET Framework 3.5 for Windows Embedded 8 Standard and Windows Server 2012

KB4480062 — Security and Quality Rollup for .NET Framework 2.0, 3.0 for Windows Server 2008

KB4480063 — Security and Quality Rollup for .NET Framework 3.5.1 for Windows Embedded Standard 7, Windows 7, and Windows Server 2008 R2

KB4480064 — Security and Quality Rollup for .NET Framework 3.5 for Windows 8.1 and Windows Server 2012 R2

KB4480070 — Security Only Update for .NET Framework 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows Embedded 8 Standard and Windows Server 2012

KB4480071 — Security Only Update for .NET Framework 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows 8.1 and Windows Server 2012 R2

KB4480072 — Security Only Update for .NET Framework 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows Embedded Standard 7, Windows 7, Windows Server 2008 R2, and Windows Server 2008

KB4480074 — Security Only Update for .NET Framework 4.5.2 for Windows 8.1 and Windows Server 2012 R2

KB4480075 — Security Only Update for .NET Framework 4.5.2 for Windows Embedded 8 Standard and Windows Server 2012

KB4480076 — Security Only Update for .NET Framework 4.5.2 for Windows Embedded Standard 7, Windows 7, Windows Server 2008 R2, and Windows Server 2008

KB4480077 — Security Only Update for .NET Framework 4 on WES09 and POSReady 2009

KB4480083 — Security Only Update for .NET Framework 3.5 for Windows Embedded 8 Standard and Windows Server 2012

KB4480084 — Security Only Update for .NET Framework 2.0, 3.0 for Windows Server 2008

KB4480085 — Security Only Update for .NET Framework 3.5.1 for Windows Embedded Standard 7, Windows 7, and Windows Server 2008 R2

KB4480086 — Security Only Update for .NET Framework 3.5 for Windows 8.1 and Windows Server 2012 R2

KB4480957 — Security Only Quality Update for Windows Server 2008

KB4480968 — Security Monthly Quality Rollup for Windows Server 2008

KB4480965 — Cumulative Security Update for Internet Explorer

KB4480972 — Security Only Quality Update for Windows Embedded 8 Standard and Windows Server 2012

KB4480975 — Security Monthly Quality Rollup for Windows Embedded 8 Standard and Windows Server 2012

KB4480979 — Adobe Flash Player update

KB4481275 — Security Update for WES09 and POSReady 2009

KB4481480 — Security and Quality Rollup for .NET Framework 3.5.1, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows Embedded Standard 7, Windows 7, and Windows Server 2008 R2

KB4481481 — Security Only Update for .NET Framework 3.5.1 for Windows Embedded Standard 7, Windows 7, and Windows Server 2008 R2

KB4481482 — Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows Embedded 8 Standard and Windows Server 2012

KB4481483 — Security Only Update for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows Embedded 8 Standard and Windows Server 2012

KB4481484 — Security and Quality Rollup for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows 8.1, Windows RT 8.1, and Windows Server 2012 R2

KB4481485 — Security Only Update for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows 8.1 and Windows Server 2012 R2

KB4481486 — Security and Quality Rollup for .NET Framework 2.0, 3.0, 4.5.2, 4.6 on Windows Server 2008

KB4481487 — Security Only Update for .NET Framework 2.0 for Windows Server 2008

KB4480056 — Cumulative Update for .NET Framework 3.5 and 4.7.2 for Windows 10 Version 1809, Windows 10 Version 1803, Windows 10 Version 1709, Windows 10 Version 1703, Windows 10 Version 1607, Windows 10 Version 1511, Windows 10 Version 1507, and Windows 10

Notes

Known Issues

Windows 10 version 1809 — KB4480116

  • Third-party applications may have difficulty authentication hotspots.

Windows 10 version 1803 — KB4480966

  • Same as Windows 10 version 1709
  • Some users may not be able to pin web links to the Start Menu or Taskbar.
  • After installing KB4467682, the cluster service may fail with 2245 (NERR_PasswordTooShort) if the Minimum Password Length policy is set to a value greater than 14 characters.KB4480966.

Windows 10 version 1709 — KB4480978

Windows 10 version 1703 — KB4480973

  • Third-party applications may have difficulty authentication hotspots.
  • Instantiation of SqlConnection can throw an exception.

Windows 10 version 1607 — KB4480961

  • Same as Windows 10 version 1709
  • After installation of KB4467691, Windows may not start on “certain” Lenovo devices with less than 8 Gigabytes of RAM.
  • After installing KB4467684, the cluster service may fail with 2245 (NERR_PasswordTooShort) if the Minimum Password Length policy is set to a value greater than 14 characters.KB4480966.
  • After installation of the update on Windows Server 2016, Outlook instant searches may fail with “Outlook cannot perform the search”.
  • System Center Virtual Machine Manager (SCVMM) managed workloads are noticing infrastructure management issues after VMM refresh as the Windows Management Instrumentation (WMI) class around network port is being unregistered on Hyper-V hosts.

Windows 8.1 — KB4480963

  • Third-party applications may have difficulty authentication hotspots.

Windows 7 — KB4480116

  • Third-party applications may have difficulty authentication hotspots.

Security advisories and updates

ADV190001 | January 2019 Adobe Flash Update

Non-security related updates

KB4090007 — Windows 10 version 1709 — Intel Microcode updates around the following products (CPUs) have been revised

KB4091663 — Windows 10 version 1703 — Intel Microcode updates around the following products (CPUs) have been revised

KB4091664 — Windows 10 version 1607 — Intel Microcode updates around the following products (CPUs) have been revised

KB890830 — Windows Malicious Software Removal Tool

Microsoft Office Updates

Microsoft released non-security updates for Office in the first week of January 2019.

The list of security updates released in January 2019 for Office is available here.

How to download and install the January 2019 security updates

windows updates january 2019

Security updates are released via Windows Update for the majority of Home systems. All Home systems are set up to check for updates automatically and download these when discovered.

Administrators may run a manual check for updates to pick up the new releases early:

  • Activate the Start Menu, e.g. by tapping on the Windows-key.
  • Type Windows Update and select the tool from the list of results.
  • Activate “check for updates” to run the manual update check.

Windows updates may be downloaded directly using third-party tools, e.g. Windows Update Minitool or wumgr, or Microsoft’s Download Center. Links to the January 2019 cumulative updates are posted below.

Direct update downloads

Microsoft publishes all cumulative security updates and other updates on the Microsoft Update Catalog website. Direct download links are listed below.

Windows 7 SP1 and Windows Server 2008 R2 SP

  • KB4480970 — 2019-01 Security Monthly Quality Rollup for Windows 7
  • KB4480960 — 2019-01 Security Only Quality Update for Windows 7

Windows 8.1 and Windows Server 2012 R2

  • KB4480963 — 2019-01 Security Monthly Quality Rollup for Windows 8.1
  • KB4480964 — 2019-01 Security Only Quality Update for Windows 8.1

Windows 10 and Windows Server 2016 (version 1607)

  • KB4480961 — 2019-01 Cumulative Update for Windows 10 Version 1607

Windows 10 (version 1703)

  • KB4480973 — 2019-01 Cumulative Update for Windows 10 Version 1703

Windows 10 (version 1709)

  • KB4480978 — 2019-01 Cumulative Update for Windows 10 Version 1709

Windows 10 (version 1803)

  • KB4480966 — 2019-01 Cumulative Update for Windows 10 Version 1803

Windows 10 (version 1809)

  • KB4480116 — 2019-01 Cumulative Update for Windows 10 Version 1809

Additional resources

Summary
Microsoft Windows Security Updates January 2019 overview
Article Name
Microsoft Windows Security Updates January 2019 overview
Description
An overview of all Microsoft security updates released for all supported versions of Microsoft Windows and other company products on January 8, 2019.
Author
Martin Brinkmann
Publisher
Ghacks Technology News
Logo
Advertisement

Related:

  • No Related Posts

Turn off Live Update

I need a solution

I have a number of machines runing Windows 7 and unmanaged Endpoint 12.x,

I need to turn off Live update but when I go into the console on each machine and into:

Change Settings,

Client Management,

Configure settings,

Live Update Tab, it is set to run ever four hours. I cannot disable “Enable automatic updates”, every time I disable this click OK and go back into the settings it is re-enabled.

Any idea how i disable this?

Andy

0

Related:

  • No Related Posts

Unable to Install

I need a solution

Hi All

I don’t mean to sound like a noob. I have been unable to install SEP on a number of our client’s machines. I get the same error across multiple machines.

The machines in question are all running Windows 7 Pro SP1, X64. I have attached a screenshot of the error I get. Kindly assist.

Thank You

Regards

0

Related:

  • No Related Posts

Live Updates every 4 Hours

I need a solution

I am running SEP 14.2 Unmanaged, and the default Live Update for Virus definitions are every 4 hours.

I usually turn off my laptop after each use.

My question is, does my laptop need to be running all the time for the Live Updates to download every 4 hours or will it continue to fetch for updates each time my machine boots up?

There is an option to select run Live Updates continuously, but that will just slow my system down and I do not want that to happen.

I would like to leave it as default which is every 4 hours, but will the Live Updates start to search for updates as soon as my machine boots up again or will I need to leave my machine ON the whole time in order for Live Update to download the virus definition files?

Thanks in advance!

0

1546788553

Related:

  • No Related Posts

OneFS and Synchronous Writes

The last article on multi-threaded I/O generated several questions on synchronous writes in OneFS. So thought this would make a useful topic to kick off the New Year and explore in a bit more detail.

OneFS natively provides a caching mechanism for synchronous writes – or writes that require a stable write acknowledgement to be returned to a client. This functionality is known as the Endurant Cache, or EC.

The EC operates in conjunction with the OneFS write cache, or coalescer, to ingest, protect and aggregate small, synchronous NFS writes. The incoming write blocks are staged to NVRAM, ensuring the integrity of the write, even during the unlikely event of a node’s power loss. Furthermore, EC also creates multiple mirrored copies of the data, further guaranteeing protection from single node and, if desired, multiple node failures.

EC improves the latency associated with synchronous writes by reducing the time to acknowledgement back to the client. This process removes the Read-Modify-Write (R-M-W) operations from the acknowledgement latency path, while also leveraging the coalescer to optimize writes to disk. EC is also tightly coupled with OneFS’ multi-threaded I/O (Multi-writer) process, to support concurrent writes from multiple client writer threads to the same file. Plus, the design of EC ensures that the cached writes do not impact snapshot performance.

The endurant cache uses write logging to combine and protect small writes at random offsets into 8KB linear writes. To achieve this, the writes go to special mirrored files, or ‘Logstores’. The response to a stable write request can be sent once the data is committed to the logstore. Logstores can be written to by several threads from the same node, and are highly optimized to enable low-latency concurrent writes.

Note that if a write uses the EC, the coalescer must also be used. If the coalescer is disabled on a file, but EC is enabled, the coalescer will still be active with all data backed by the EC.

So what exactly does an endurant cache write sequence look like?

Say an NFS client wishes to write a file to an Isilon cluster over NFS with the O_SYNC flag set, requiring a confirmed or synchronous write acknowledgement. Here is the sequence of events that occur to facilitate a stable write.

1) A client, connected to node 3, begins the write process sending protocol level blocks.



ec_1.png



4KB is the optimal block size for the endurant cache.

2) The NFS client’s writes are temporarily stored in the write coalescer portion of node 3’s RAM. The Write Coalescer aggregates uncommitted blocks so that the OneFS can, ideally, write out full protection groups where possible, reducing latency over protocols that allow “unstable” writes. Writing to RAM has far less latency that writing directly to disk.

3) Once in the write coalescer, the endurant cache log-writer process writes mirrored copies of the data blocks in parallel to the EC Log Files.



ec_2.png

The protection level of the mirrored EC log files is the same as that of the data being written by the NFS client.

4) When the data copies are received into the EC Log Files, a stable write exists and a write acknowledgement (ACK) is returned to the NFS client confirming the stable write has occurred.



ec_3.png



The client assumes the write is completed and can close the write session.

5) The write coalescer then processes the file just like a non-EC write at this point. The write coalescer fills and is routinely flushed as required as an asynchronous write via to the block allocation manager (BAM) and the BAM safe write (BSW) path processes.

6) The file is split into 128K data stripe units (DSUs), parity protection (FEC) is calculated and FEC stripe units (FSUs) are created.



ec_4.png

7) The layout and write plan is then determined, and the stripe units are written to their corresponding nodes’ L2 Cache and NVRAM. The EC logfiles are cleared from NVRAM at this point. OneFS uses a Fast Invalid Path process to de-allocate the EC Log Files from NVRAM.



ec_5.png

8) Stripe Units are then flushed to physical disk.

9) Once written to physical disk, the data stripe Unit (DSU) and FEC stripe unit (FSU) copies created during the write are cleared from NVRAM but remain in L2 cache until flushed to make room for more recently accessed data.



ec_6.png

As far as protection goes, the number of logfile mirrors created by EC is always one more than the on-disk protection level of the file. For example:

File Protection Level

Number of EC Mirrored Copies

+1n

2

2x

3

+2n

3

+2d:1n

3

+3n

4

+3d:1n

4

+4n

5

The EC mirrors are only used if the initiator node is lost. In the unlikely event that this occurs, the participant nodes replay their EC journals and complete the writes.

If the write is an EC candidate, the data remains in the coalescer, an EC write is constructed, and the appropriate coalescer region is marked as EC. The EC write is a write into a logstore (hidden mirrored file) and the data is placed into the journal.

Assuming the journal is sufficiently empty, the write is held there (cached) and only flushed to disk when the journal is full, thereby saving additional disk activity.

An optimal workload for EC involves small-block synchronous, sequential writes – something like an audit or redo log, for example. In that case, the coalescer will accumulate a full protection group’s worth of data and be able to perform an efficient FEC write.

The happy medium is a synchronous small block type load, particularly where the I/O rate is low and the client is latency-sensitive. In this case, the latency will be reduced and, if the I/O rate is low enough, it won’t create serious pressure.

The undesirable scenario is when the cluster is already spindle-bound and the workload is such that it generates a lot of journal pressure. In this case, EC is just going to aggravate things.

So how exactly do you configure the endurant cache?

Although on by default, setting the efs.bam.ec.mode sysctl to value ‘1’ will enable the Endurant Cache:

# isi_for_array –s isi_sysctl_cluster efs.bam.ec.mode=1

EC can also be enabled & disabled per directory:

# isi set -c [on|off|endurant_all|coal_only] <directory_name>

To enable the coalescer but switch of EC, run:

# isi set -c coal_only

And to disable the endurant cache completely:

# isi_for_array –s isi_sysctl_cluster efs.bam.ec.mode=0

A return value of zero on each node from the following command will verify that EC is disabled across the cluster:

# isi_for_array –s sysctl efs.bam.ec.stats.write_blocks efs.bam.ec.stats.write_blocks: 0

If the output to this command is incrementing, EC is delivering stable writes.

As mentioned previously, EC applies to stable writes. Namely:

  • Writes with O_SYNC and/or O_DIRECT flags set
  • Files on synchronous NFS mounts

When it comes to analyzing any performance issues involving EC workloads, consider the following:

  • What changed with the workload?
  • If upgrading OneFS, did the prior version also have EC enable?
  • If the workload has moved to new cluster hardware:
  • Does the performance issue occur during periods of high CPU utilization?
  • Which part of the workload is creating a deluge of stable writes?
  • Was there a large change in spindle or node count?
  • Has the OneFS protection level changed?
  • Is the SSD strategy the same?

Disabling EC is typically done cluster-wide and this can adversely impact certain workflow elements. If the EC load is localized to a subset of the files being written, an alternative way to reduce the EC heat might be to disable the coalescer buffers for some particular target directories, which would be a more targeted adjustment. This can be configured via the isi set –c off command.

One of the more likely causes of performance degradation is from applications aggressively flushing over-writes and, as a result, generating a flurry of ‘commit’ operations. This can generate heavy read/modify/write (r-m-w) cycles, inflating the average disk queue depth, and resulting in significantly slower random reads. The isi statistics protocol CLI command output will indicate whether the ‘commit’ rate is high.

It’s worth noting that synchronous writes do not require using the NFS ‘sync’ mount option. Any programmer who is concerned with write persistence can simply specify an O_FSYNC or O_DIRECT flag on the open() operation to force synchronous write semantics for that fie handle. With Linux, writes using O_DIRECT will be separately accounted-for in the Linux ‘mountstats’ output. Although it’s almost exclusively associated with NFS, the EC code is actually protocol-agnostic. If writes are synchronous (write-through) and are either misaligned or smaller than 8KB, they have the potential to trigger EC, regardless of the protocol.

The endurant cache can provide a significant latency benefit for small (eg. 4K), random synchronous writes – albeit at a cost of some additional work for the system.

However, it’s worth bearing the following caveats in mind:

  • EC is not intended for more general purpose I/O.
  • There is a finite amount of EC available. As load increases, EC can potentially ‘fall behind’ and end up being a bottleneck.
  • Endurant Cache does not improve read performance, since it’s strictly part of the write process.
  • EC will not increase performance of asynchronous writes – only synchronous writes.

Related:

  • No Related Posts