Avaya & 3rd Party Certificates running on a Server Edition

Ok! So, post number 1 and what a place to start! This is something I have done battle with over and over again. Ive read and re-read every version of the security guidelines documents from R9.1, when 3rd party certificate support came in.

Believe it or not, it all started with me wanting/needing to have a more secure way to connect remote H323 handsets without the need for a VPN or Oracle SBC (yes a SIP B2BUA with a H323 gatekeeper proxy) As R9.1 was out and this brought with it, the support for 3rd party certificates and TLS on the 96×1 ranges. I knew my task was clear.

I quickly discovered that 1. H323 handsets cant be secured with a 3rd party certificate as they wont sign on IP. 2. The “use separate certificate for telephony ” in security settings, is not very good, doesn’t load the cert correctly and if you look at an R11 system. ITs now ‘for separate SIP telephony’, not all telephony.

It wasn’t until after months and months of failed attempts, that I found a fix on the Avaya support pages. Not for this, don’t worry Im not just going to recycle T4’s work. This was a fix for 3rd party certificates, not being automatically distributed to all the apps and only the IP Offices home page. It was down to the lack of a file. This got me thinking, could I deploy my 3rd party certs with said file in place. Let them distribute out to everything, but then remove the file and upload another certificate, in this case an Avaya signed one. The answer was yes, this did work, in a backwards, bush fix style workaround.

I will now try to guide you through the process for this. Getting you setup to run a H323 over TLS with Avaya Certs and a 3rd party Certificate, like a wildcard certificate, to secure the web apps. If you have a UCC certificate for you SIP devices and your not using an SBC like me, in theory you could apply that in security settings for separate telephony. Though I have not tested this one yet, so cant make any promises.

Below is a list of things you will need for this.

    • Putty
    • WinSCP
    • Manager
    • A 3rd party certificate
    • A separate HTTP file server
    • Coffee or Tea

Step 1

Connect to your Server Editions platform view on port 7071. Make you way to the settings tab and the general tab within that. Scroll down, about half way, until you reach the certificates section. Now we need to create the certificate details that our Avaya CA will sign. We can add the FQDN to this but we must include the internal and external IP address that the H323 phones will use as their gatekeeper. Below is an example of the format to use. Click the download PEM option at the top to get the root-ca then, once happy click the regenerate button at the bottom and let the certificate populate.

Step 2 

Once the certificate has populated across the IP Office and all the applications. We can connect in a retrieve the ID certificate, CA certificate and the private key. To do this, open putty and SSH to your IP Office. Login with your Administrator account, type admin, enter the Administrator credentials again, then type root. Put in your root password (set at ignition – security password) your prompt in the linux should now show as root. Using the below two commands, navigate to the Certificates store on the server and view the files.

cd /opt/Avaya/Certs 

ls -la

Step 3

Copy these files to your home directory using the below command.

cp *.pem /home/Administrator/

Then use this command to change the privileges those files have.

chmod 777 /home/Administror/*.pem

Now open WinSCP and connect to the IP Offie using your Administrator login and the servers IP address. You will find that this takes you directly to the /home/Administrator directory. We should see our .pem files. Copy all of these to a folder on your machine. We don’t really need all of them, its just easier. Make sure to copy the root-ca.pem that you downloaded earlier, to the root of your HTTP file server, make sure its named Root-CA.pem

Step 4 

Now we have our IP Office certificate in place, We can setup the rest of the changes needed and test that the handsets work over TLS. The first thing we need to do is to make sure that the CRAFT password has been updated and that we have updated our 46xxsettings.txt file on the HTTP server, so that the phone knows what certificate to take etc.

To change the CRAFT password, Goto no user in the config and add a source number of:


make a note of the code you use, its what you have to type to enter the craft menu, not just when interrupting a boot. This will only help on an auto gen 46xx. If you have one already, just check the H323 install guide for what is needed.

Make sure the TLS settings section of your 46xxsettings appears as below. The easiest way is to use the auto generated 46xx. Copy and paste in your own file, changing the bits you need. Don’t worry if  there is extra after the root-ca bit. just remove that part. We don’t need it. We dont need to worry about the Auto Gen 46xx as we will be pointing the phones to the external HTTP server.


Now as long as your phone is pointing to your IP Office and HTTP server, the above has been completed and that you have TLS enabled on the IP Office. I have assumed till now that you do. You should now have a 96×1 using TLS.

Step 5

As we now have working phones using TLS. Its only fitting, that we temporarily break them 🙂 Its a very simple step, we just add our 3rd party certificate to the server and let it distribute out to all the applications.

Don’t worry!! if you don’t know how to add or prepare a 3rd party certificate for the IP Office. We have you covered. Just go to our post on doing exactly that here.

Step 6

Once we have recovered from the shock of everything stopping and the phones logging out. We can move on to making everything work amazingly. To do so we need to repeat a couple of bits from step 2. and navigate to /opt/Avaya/certs as root. Run ls -la to confirm the hidden .auto file is still appearing.

This is the magical part, the .auto file has nothing in it and doesn’t do anything. It just needs to be there. If its there, the certificate will be distributed to everything. If its not, then the cert is only applied to the IP Office. Because of this, we want to remove the file, stopping the distribution of any other new certificates. The command for this is below.

rm .auto

If for any reason, you need to reverse this. Just go to this directory and run

touch .auto

Your certs will start distributing again!

Step 7

Open up Manager and login to your server edition. Once in, goto File> Advanced>Security Settings

Within the main Security Settings, under System, go to the Certificates tab. Before we start, we need to do a tiny bit of prep work. Open up the cert.pem that you copied from the IPO in notepad. do a ctrl+A then ctrl+C. Open another blank notepad and paste in your copied cert.pem contents. Now open the root-ca.pem that you downloaded from platform view and copy all the contents. Paste all of this into your new notepad doc. do not leave any spaces from your first addition of the cert.pem.

Now back in Manager, you should see an option to set a new certificate. chose this option followed by the paste from file option. This will bring up a popup with two tabs. One for the certificate and one for the key. Select all of the text in your notepad file and paste it all into the cert box. Move into the key tab, open the key.pem copied earlier and copy/paste the contents of this into the key tab.  Click ok and wait with baited breath. After a few seconds, your original IP Office certificate should be set.

Step 8 

Confirmation time. Browse to your Server Editions FQDN on any port option :7070, :7071 or :9443 and the relevant directory. You should still have a green HTTPS lock and if you check the cert, it will be your 3rd party certificate. Now reboot your H323 phone and screem with joy as it connects to the IP Office without the failed TLS connection beep of doom and logs in with all that secure glory.



Leave a Reply

Your email address will not be published. Required fields are marked *