Notes / Domino by Daniel Nashed 

Domino V14 backup for notes.ini  

By Daniel Nashed | 6/1/23 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino backup is around since 12.0 and it got improvements in every release. There are not many current AHA ideas for Domino Backup & Restore. One smaller feature you can see in EAP1 is the backup of the notes.ini.

How to use Domino OTS on Kubernetes to import an existing TLS Certificate  

By Daniel Nashed | 5/30/23 12:05 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino One Touch Setup has been designed with flexibility in mind, with special focus on getting a server up in a secure way. On Docker you can just mount PEM files into the container. On Kubernetes TLS Certificates and Keys are stored in secrets. Personally I am not a big fan of storing PEM files on disk. But you could at least set a password on the PEM file you import. Here is a basic example how to create a secret on K8s and reference it in OTS. Even the simple environment variable setup supports the security settings for CertMgr. Of course the same functionality is also available with the more flexible JSON based configuration.

Importing trusted MicroCA Roots for a Nomad Lab environment  

By Daniel Nashed | 5/29/23 12:39 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Yesterday I worked on a lab configuration based on Windows Sandbox, Domino and Nomad Web. The biggest challenge is to have a trusted certificate for Nomad Web. Nomad Server running with the Micro CA A Nomad Server can use Domino CertMgr Micro CA Certs. But the root is not trusted in your browser. I took a closer look and came up with a simple solution. which makes the import dramatically easier. No more searching for the right trust store and handling PEM files manually.

Tuning Domino Servers for TLS sessions  

By Daniel Nashed | 5/24/23 2:13 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

My previous post was mainly about HTTP traffic and I mentioned TLS/SSL don't use the maximum number of connections settings, because they have a SSL/TLS session. Establishing a new TLS session has significant overhead! And you have to make sure in any application, that those sessions are cached and resumed. I revisited a blog post from 2012 where I explained a fix, which went into 8.5.3. And was enabled in 8.5.4 by default (which turned into the 9.0 release when shipped as far I recall). There was an issue with the session cache and a new cache had been implemented in 8.5.3. Today the new cache is the default and SSL_USE_ADDSESSION2=1 does not exist any more.

TLS/SSL Cipher Troubleshooting  

By Daniel Nashed | 4/10/23 9:35 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Every Domino release adds more TLS ciphers to the weak list to ensure poper security. We can expect the next versions also to have less ciphers available. Domino ensures for clients and servers, that the list of ciphers provided is safe. In addition the default behavior is that the server decides the order of ciphers to pick. And only allows secure renegotiation to prevent the client to pick a less secure cipher. Usually this doesn't cause a lot of trouble for inbound connections. Modern browsers support modern ciphers. But outgoing connections for LDAP and ICAP could be a challenge. I had to look into an ICAP connection problem this week. To demonstrate how the TLS handshake works, I wrote a small OpenSSL demo program in C. This turned into a quite flexible troubleshooting tool over the weekend.

Picking the right Linux Distribution  

By Daniel Nashed | 4/5/23 10:42 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

There isn't one best or right distribution in general. There are many variations. For example Redhat/CentOS based platforms have flavors like Rocky and Alma Linux. The HCL Domino community project looks into many possible combinations as the base image --> https://opensource.hcltechsw.com/domino-container/concept_environments/ There are basically three different main flavors with different toppings: - Redhat/CentOS based (with yum and dnf in later versions to manage packages) - Ubuntu/Debian (with apt to manage packages) - SUSE Enterprise/Leap etc. (with zypper to manage packages) The right distribution is really depending on your needs.

Nomad Web server connection options  

By Daniel Nashed | 1/25/23 7:30 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

Nomad Web is a modern HCL client offering in form of a Progressive Web Application (PWA) running in your web browser. In addition to Windows or Mac, it also works on Ubuntu and other Linux distributions! So there is finally a client offering for Linux clients again! The Nomad Web application is installed on a server providing the required files for download. Those files can be stored on a SafeLinx or Domino/Nomad Web server.

Windows Sandbox - A feature you should know  

By Daniel Nashed | 1/23/23 2:20 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The sandbox can be a very useful tool for many different situations. I am often using it for Domino server or client install tests. But there are many other use cases including training environments etc. It's a full throw away sandbox environment recreated every time you start it. The only limitation is that you can't reboot the Windows for example after a software update. But even installing the Windows re-distributable run-time package does not require a boot. Most applications like Notes/Domino install it on their own. I needed it to test my own applications. But there is an easy way to download and silent install it:

NGINX TCP Stream with SNI support. More than helpful for lab environments  

By Daniel Nashed | 1/23/23 2:15 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In production you usually want centralized certificate handling and off-loading TLS termination to a load-balancer. I posted scripts to have NGINX realod certs automatically from Domino CertMgr via HTTPS to leverage Domino's Let's Encrypt implementation. But sometimes you really want all your servers directly exposed over TLS. For example in a lab environment with limited resources and only one IP, you might want to still have each of the hosts expose their services on their own.

How to get the error message for a Notes error code  

By Daniel Nashed | 12/13/22 1:34 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Sometimes scripts or Domino server commands only return an error code and you would like to know the error message. There is an easy way to get the error message back from a server command. "show message [module] In most cases you don't need server tasks specific error messages and just use the decimal error code.

Restic – Command Line Tool supporting Windows VSS  

By Daniel Nashed | 11/24/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that Domino 12.0.2 is has a native VSS Writer, we can look into new interesting integrations. In my session at SUTOL conference this week, I showed a first version of a Restic integration for Domino 12.0.2 via VSS. Restic is a very interesting application (https://restic.net) It's a single binary written in GO And uses a approach like Borg Backup uses. But in contrast to Borg Backup it has full Windows support. This includes VSS Writer + AutoRecovery support! It is Open Source, efficient, flexible & secure. And very simple to setup & use!

Docker cp with permissions and owner change  

By Daniel Nashed | 10/11/22 2:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

By the default the ownership of a file copied into a running container is always root:root. Depending on how you want to use the copied file, this ownership isn't what you want. You can't pass user/group or permissions to the docker cp command. Changing the owner or mode would need root permissions inside the container. Containers usually run with an unprivileged application user. For Domino this is notes:notes with the IDs 1000:1000.

Domino 12.0.2 One Touch setup with Let’s Encrypt certificates  

By Daniel Nashed | 9/15/22 3:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

One Touch setup (OTS) is quite powerful tool. But sometimes you need to know exactly what happens and combine functionality to make best use of it. With Domino 12.0.2 OTS creates certstore.nsf automatically and you can let it create a MicroCA for you. But what if you want to use a Let's Encrypt certificate instead? There is a quite simple way to just find and update the existing document with a appConfiguration. And if you specify notes.ini CertMgr_ACCEPT_TOU=1 the ACME account license agreement will be automatically accepted (already part of 12.0.0).

Download certificate chain without OpenSSL  

By Daniel Nashed | 9/12/22 3:17 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Usually OpenSSL is the tool of choice for all type of certificate operations. But what if no OpenSSL command line is available? Like in a Domino container where you can't install software? After some research, I came up with the keytool, which is part of the JVM Domino ships.

K3s, Podman and a registry  

By Daniel Nashed | 9/5/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Rancher Desktop is a great all-in-one desktop environment. When running it with the Docker back-end you have all in one environment for development and run-time. For a server, K3s (https://k3s.io) is my platform of choice. It is production ready and easy to deploy. For Kubernetes, you always need a registry to pull images. As soon you need custom images, you will need a registry to upload and download your image. K3s allows you to configure private registries. You could use any registry. I am just running the registry Docker image on Podman in my environment.

Domino Community Image - New Nomad Server install option  

By Daniel Nashed | 7/30/22 10:22 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

The Nomad server is a new offering to directly add Nomad support to your Domino server instead of using a SafeLinx server. Recently I added a SafeLinx container to the Domino community project. Now I am adding the Nomad Server to the Domino image as a new build options.

Domino Container automation testing  

By Daniel Nashed | 7/25/22 12:20 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Containers are not only a good way to run Domino. It is also the perfect environment for automation testing. Domino 12 introduced OneTouch Setup to automate deployments, which also lets you create reproducible Domino test server scenarios. As a starting point I am building an automation test for the Domino image itself. So in future for every commit on the Git repository I can run automation to ensure the image works. The test automation can be used in your own environment as well after an image built in your environment. It can be also be extended for your own application testing.

Customizing Domino Backup mail notifications  

By Daniel Nashed | 7/22/22 6:42 PM | Infrastructure - Notes / Domino | Added by Oliver Busse

Domino Backup offers to send e-mails depending on the status of your backup. By default you are getting an e-mail in case of error or warning. I am rarely getting error messages from my servers. In this case here I updated my server to a new kernel and ZFS drivers failed to build.

Nomad Server 12.0.2 on Linux just works  

By Daniel Nashed | 7/20/22 10:32 PM | Infrastructure - Notes / Domino | Added by Oliver Busse

The Nomad Server is a small component, you install on your Domino server. And it is bundled with the Nomad Web files. So it is a all-in one server add-on solution. The installation sounds more complicated then it is. It's really simple to install. And I am thinking about making it an install option for the Domino community container image.

Why run Domino in a container today  

By Daniel Nashed | 7/20/22 2:03 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

As my of you know, I am a big fan of running Domino and other applications in a container. This can be a classical Docker/Podman deployment or K8s. Containers might not be good for everyone. But a lot of software is available in a "Docker image", which can run in multiple environments. Domino's main deployment model will not change to Docker.

Domino 12 Restore point in time  

By Daniel Nashed | 7/15/22 1:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino Backup can restore databases point in time! For other backup applications this functionality is usually only available with archive transaction log. But with circular translog or linear translog mode, the most current backup should have all the translogs available to recover point in time as well. Domino Restore allows you to restore point in time in that case. Even it is not guaranteed that the translogs are still there, this can be still a good configuration if you have servers without dramatic load.

Linux shell scripts: Difference between "set" and "env" -- fixed the Domino start script  

By Daniel Nashed | 7/11/22 4:28 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The Domino start script has always been using "set" to list the environment variables, before running the sever. This information can be important to understand the environment passed to your Domino server at start-up. It turns out that there had been a change over time, which causes much more information to be listed, then just the environment variables.

Domino ZFS Snapshot Backup  

By Daniel Nashed | 6/20/22 2:05 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

ZFS is one of my favorite file-systems. And I posted before about using it as a backup target. The integration is pretty simple with Domino backup, because it is a simple file backup. Now that we have the new VSS Writer for Domino 12.0.2 on Windows, it is time to look into ZFS snapshots.

Recovering a lost Domino server notes.ini quickly  

By Daniel Nashed | 5/26/22 1:39 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

On Linux by default the notes.ini is in the data directory. On Windows it is per default in the binary directory. You could move it to the data directory, which would make sense from backup point of view in many cases anyway. But what if you have it in the program directory and install a new major version where you get rid of all your binaries as a best practice?

SHA512 is faster then SHA256  

By Daniel Nashed | 5/13/22 6:12 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Quite interesting results.. I have been looking into different hash algorithms to see the overhead today. It turns out that SHA256 is the slowest option and SHA1 is the winner. But it is interesting, that SHA384/SHA512 are also faster then SHA256

Faster Domino server restart   

By Daniel Nashed | 5/13/22 6:11 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino waits for 10 seconds after shutdown before are restart for some legacy reason. There is a notes.ini variable to reduce the number of seconds. I tested with Domino 12.0.x that it can be reduced to 1 second.

K8s Certificate Manager with Let’s Encrypt  

By Daniel Nashed | 5/5/22 4:41 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino certificate manager works like a charm and is the best option for native Domino 12 certificate management. But in a K8s environment you might want to better have certificates deployed outside Domino in front of your Domino K8s service. Mostly you will use a so called Ingress controller, which offloads your TLS traffic. I took a look into https://cert-manager.io/docs/concepts/certificate last night. It turned out the issues I ran into only occurred because of a messed up k3s installation. After I re-created my server, I was ready to go in minutes.

Full Domino Fail2Ban Integration  

By Daniel Nashed | 3/21/22 3:45 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This week we tried to get Domino 12.0.1 IP based blocking working for our DNUG server. It turns out to work great for the HTTP protocol. But our Sametime server got blocked on port LDAPS when verifying log-in information via LDAPS.

HCL Domino Docker Container - Moved to a new home!  

By Daniel Nashed | 3/16/22 3:25 PM | Infrastructure - Notes / Domino | Added by Oliver Busse

The Domino Docker project was started by Thomas Hampel with Domino 9.0.1 at IBM. He introduced me to the project to contribute my Domino start script. I wrote a lot of code and added a lot of functionality since then. Thomas and I did many presentations together and it always was and will continue to be one of our favorite projects!

SpamGeek works -- O365 finally got blocked sending too much spam  

By Daniel Nashed | 3/15/22 5:29 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Ohhh ... I was wondering that some mails did not reach me any more. And I took a look into my SpamGeek logs. Over time the SPAM score of the domain got higher and now it was blocking also other senders from that domain. It got higher because the words I added to the negative list. And the ratio between good and bad mail changed..