Database frequently stop connecting after upgrade 19c

We’re facing the issue database frequently stop connecting after upgrade to 19c (19.3). this database which is running on windows server 2016.

Errors in listener.log file

TNS-12518: TNS:listener could not hand off client connection
TNS-12546: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00516: Permission denied
64-bit Windows Error: 5: Input/output error

Core dump should created on somewhere in your database server locations.

Solution :

Apply database 19.7 or latest patches. the issue will be fixed

ORA-07445: exception encountered: core dump [ph2sf_scan_fields()+594] [ACCESS_VIOLATION] [ADDR:0x8] [PC:0x7FFA9B633C52] [UNABLE_TO_READ] []

We’re getting the below error while running utlrp.sql script after we applied 19.9 patch to database 19.3 which is running on windows server 2016.

2021-01-05T23:16:39.538245+03:00
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0x8] [PC:0x7FFA9B633C52, ph2sf_scan_fields()+594]
Errors in file D:\APP\ORACLE\diag\rdbms\ezymir\ezymir\trace\ezymir_j000_94264.trc (incident=136519):
ORA-07445: exception encountered: core dump [ph2sf_scan_fields()+594] [ACCESS_VIOLATION] [ADDR:0x8] [PC:0x7FFA9B633C52] [UNABLE_TO_READ] []
Incident details in: D:\APP\ORACLE\diag\rdbms\ezymir\ezymir\incident\incdir_136519\ezymir_j000_94264_i136519.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

Initially we thinking that its seems to be bug. But After we analyzed the trace file, there is one package body having issue for compilations.

Trace file :

name=XXXXX.CLK_CLM_PROCESS
Symbol file D:\app\WINDOWS.X64_193000_db_home\BIN\orasql19.SYM does not match binary.
Symbol TimeStamp=5cef5a1d, Module TimeStamp=5f8f4599 are different
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0x8] [PC:0x7FFA9B633C52, ph2sf_scan_fields()+594]
Encountered exception while getting args for function:0x00007FFA9B685811
Symbol file D:\app\WINDOWS.X64_193000_db_home\BIN\orasql19.SYM does not match binary.
Symbol TimeStamp=5cef5a1d, Module TimeStamp=5f8f4599 are different
Encountered exception while getting args for function:0x00007FFA9B690D40
Encountered exception while getting args for function:0x00007FFA9B685811

*** 2021-01-05T23:16:40.272618+03:00
2021-01-05T23:16:40.272618+03:00
Incident 136519 created, dump file: D:\APP\ORACLE\diag\rdbms\ezymir\ezymir\incident\incdir_136519\ezymir_j000_94264_i136519.trc
ORA-07445: exception encountered: core dump [ph2sf_scan_fields()+594] [ACCESS_VIOLATION] [ADDR:0x8] [PC:0x7FFA9B633C52] [UNABLE_TO_READ] []

Solution :

We found that we have master schema there . this schema having lot of objects. those objects having read/write permission to other schemas. But we have missed some permission to some particular schema when we’re creating or importing. once we provided the particular privilege to the schema. we can able to compile the packages using utlrp.sql as well.

permission given like

GRANT ALTER, DELETE, INDEX, INSERT, REFERENCES, SELECT, UPDATE, ON COMMIT REFRESH, QUERY REWRITE, READ, DEBUG, FLASHBACK ON MASTERSCHEMA.table to xxxxx;

antivirus running on oracle database server

Read this article

https://pmdba.wordpress.com/2020/02/11/antivirus-for-my-database-server/

How To Configure Anti-Virus On Oracle Database Server (Doc ID 782354.1)

TNS-12557 TNS-12560 TNS-00527

While doing upgrade to 19c, dbua getting failed during netca listener creation in windows with following error

TNS-12557: TNS:protocol adapter not loadable
TNS-12560: TNS:protocol adapter error
TNS-00527: Protocol Adapter not loadable

Solution :

Before you start DBUA, please check the windows oracle_home & tns_admin environment set with old home & tns admin path. remove the old path from environment variables and start with 19c oracle home & path.

Oracle database upgrade from 12c to 19c

Install 19c software

If you get the error “the program can’t start because api-msi-win-crt-runtime-l1-1-0.dll is missing from your computer”

Download and install

https://www.microsoft.com/en-in/download/details.aspx?id=48145

Start installation

C:\Users\mazar>set ORACLE_HOME=E:\app\cmsdb\product\12.2.0\dbhome_1

C:\Users\mazar>set PATH=E:\app\cmsdb\product\12.2.0\dbhome_1\bin

C:\Users\mazar>set ORACLE_SID=cmsqadb

C:\Users\mazar>E:\app\cmsdb\product\12.2.0\dbhome_1\jdk\bin\java -jar E:\app\WINDOWS.X64_193000_db_home\rdbms\admin\preupgrade.jar TEXT TERMINAL
Report generated by Oracle Database Pre-Upgrade Information Tool Version
19.0.0.0.0 Build: 1 on 2020-11-05T13:13:58

Upgrade-To version: 19.0.0.0.0

=======================================

Status of the database prior to upgrade

  Database Name:  CMSQADB
 Container Name:  cmsqadb
   Container ID:  0
        Version:  12.2.0.1.0
 DB Patch Level:  No Patch Bundle applied
     Compatible:  12.2.0
      Blocksize:  8192
       Platform:  Microsoft Windows x86 64-bit
  Timezone File:  26

Database log mode: NOARCHIVELOG
Readonly: FALSE
Edition: EE

Oracle Component Upgrade Action Current Status
—————- ————– ————–
Oracle Server [to be upgraded] VALID
JServer JAVA Virtual Machine [to be upgraded] VALID
Oracle XDK for Java [to be upgraded] VALID
Real Application Clusters [to be upgraded] OPTION OFF
Oracle Workspace Manager [to be upgraded] VALID
OLAP Analytic Workspace [to be upgraded] VALID
Oracle Label Security [to be upgraded] VALID
Oracle Database Vault [to be upgraded] VALID
Oracle Text [to be upgraded] VALID
Oracle XML Database [to be upgraded] VALID
Oracle Java Packages [to be upgraded] VALID
Oracle Multimedia [to be upgraded] VALID
Oracle Spatial [to be upgraded] VALID
Oracle OLAP API [to be upgraded] VALID

==============

BEFORE UPGRADE

REQUIRED ACTIONS
================

1. (AUTOFIXUP) Empty the RECYCLEBIN immediately before database upgrade. The database contains 11 objects in the recycle bin. The recycle bin must be completely empty before database upgrade. RECOMMENDED ACTIONS
2. Run 12.2.0.1.0 $ORACLE_HOME/rdbms/admin/utlrp.sql to recompile invalid
   objects. You can view the individual invalid objects with SET SERVEROUTPUT ON; EXECUTE DBMS_PREUP.INVALID_OBJECTS; 631 objects are INVALID. There should be no INVALID objects in SYS/SYSTEM or user schemas before
   database upgrade.
3. (AUTOFIXUP) Gather stale data dictionary statistics prior to database
   upgrade in off-peak time using: EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS; Dictionary statistics do not exist or are stale (not up-to-date). Dictionary statistics help the Oracle optimizer find efficient SQL
   execution plans and are essential for proper upgrade timing. Oracle
   recommends gathering dictionary statistics in the last 24 hours before
   database upgrade. For information on managing optimizer statistics, refer to the 12.2.0.1
   Oracle Database SQL Tuning Guide. INFORMATION ONLY
4. To help you keep track of your tablespace allocations, the following
   AUTOEXTEND tablespaces are expected to successfully EXTEND during the
   upgrade process. Min Size Tablespace Size For Upgrade
   ———- ———- ———–
   SYSTEM 938 MB 1032 MB Minimum tablespace sizes for upgrade are estimates.
5. Check the Oracle Backup and Recovery User’s Guide for information on how
   to manage an RMAN recovery catalog schema. If you are using a version of the recovery catalog schema that is older
   than that required by the RMAN client version, then you must upgrade the
   catalog schema. It is good practice to have the catalog schema the same or higher version
   than the RMAN client version you are using. ORACLE GENERATED FIXUP SCRIPT All of the issues in database CMSQADB
   which are identified above as BEFORE UPGRADE “(AUTOFIXUP)” can be resolved by
   executing the following SQL>@E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade/preupgrade_fixups.sql

=============

AFTER UPGRADE

REQUIRED ACTIONS
================
None

RECOMMENDED ACTIONS
===================

6. Upgrade the database time zone file using the DBMS_DST package. The database is using time zone file version 26 and the target 19 release
   ships with time zone file version 32. Oracle recommends upgrading to the desired (latest) version of the time
   zone file. For more information, refer to “Upgrading the Time Zone File
   and Timestamp with Time Zone Data” in the 19 Oracle Database
   Globalization Support Guide.
7. To identify directory objects with symbolic links in the path name, run
   $ORACLE_HOME/rdbms/admin/utldirsymlink.sql AS SYSDBA after upgrade.
   Recreate any directory objects listed, using path names that contain no
   symbolic links. Some directory object path names may currently contain symbolic links. Starting in Release 18c, symbolic links are not allowed in directory
   object path names used with BFILE data types, the UTL_FILE package, or
   external tables.
8. (AUTOFIXUP) Gather dictionary statistics after the upgrade using the
   command: EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS; Oracle recommends gathering dictionary statistics after upgrade. Dictionary statistics provide essential information to the Oracle
   optimizer to help it find efficient SQL execution plans. After a database
   upgrade, statistics need to be re-gathered as there can now be tables
   that have significantly changed during the upgrade or new tables that do
   not have statistics gathered yet.
9. Gather statistics on fixed objects after the upgrade and when there is a
   representative workload on the system using the command: EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS; This recommendation is given for all preupgrade runs. Fixed object statistics provide essential information to the Oracle
   optimizer to help it find efficient SQL execution plans. Those
   statistics are specific to the Oracle Database release that generates
   them, and can be stale upon database upgrade. For information on managing optimizer statistics, refer to the 12.2.0.1
   Oracle Database SQL Tuning Guide. ORACLE GENERATED FIXUP SCRIPT All of the issues in database CMSQADB
   which are identified above as AFTER UPGRADE “(AUTOFIXUP)” can be resolved by
   executing the following SQL>@E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade/postupgrade_fixups.sql

==================

PREUPGRADE SUMMARY

E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade\preupgrade.log
E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade\preupgrade_fixups.sql
E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade\postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade:

Log into the database and execute the preupgrade fixups
@E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade\preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade\postupgrade_fixups.sql

Preupgrade complete: 2020-11-05T13:13:58

C:\Users\mazar>

SQL> purge recyclebin;

Recyclebin purged.

SQL>

SQL> @E:\app\cmsdb\product\12.2.0\dbhome_1\rdbms\admin\utlrp.sql

TIMESTAMP

COMP_TIMESTAMP UTLRP_BGN 2020-11-05 13:28:14

DOC> The following PL/SQL block invokes UTL_RECOMP to recompile invalid
DOC> objects in the database. Recompilation time is proportional to the
DOC> number of invalid objects in the database, so this command may take
DOC> a long time to execute on a database with a large number of invalid
DOC> objects.
DOC>
DOC> Use the following queries to track recompilation progress:
DOC>
DOC> 1. Query returning the number of invalid objects remaining. This
DOC> number should decrease with time.
DOC> SELECT COUNT() FROM obj$ WHERE status IN (4, 5, 6); DOC> DOC> 2. Query returning the number of objects compiled so far. This number DOC> should increase with time. DOC> SELECT COUNT() FROM UTL_RECOMP_COMPILED;
DOC>
DOC> This script automatically chooses serial or parallel recompilation
DOC> based on the number of CPUs available (parameter cpu_count) multiplied
DOC> by the number of threads per CPU (parameter parallel_threads_per_cpu).
DOC> On RAC, this number is added across all RAC nodes.
DOC>
DOC> UTL_RECOMP uses DBMS_SCHEDULER to create jobs for parallel
DOC> recompilation. Jobs are created without instance affinity so that they
DOC> can migrate across RAC nodes. Use the following queries to verify
DOC> whether UTL_RECOMP jobs are being created and run correctly:
DOC>
DOC> 1. Query showing jobs created by UTL_RECOMP
DOC> SELECT job_name FROM dba_scheduler_jobs
DOC> WHERE job_name like ‘UTL_RECOMP_SLAVE_%’;
DOC>
DOC> 2. Query showing UTL_RECOMP jobs that are running
DOC> SELECT job_name FROM dba_scheduler_running_jobs
DOC> WHERE job_name like ‘UTL_RECOMP_SLAVE_%’;
DOC>#

PL/SQL procedure successfully completed.

TIMESTAMP

COMP_TIMESTAMP UTLRP_END 2020-11-05 13:34:32

DOC> The following query reports the number of invalid objects.
DOC>
DOC> If the number is higher than expected, please examine the error
DOC> messages reported with each object (using SHOW ERRORS) to see if they
DOC> point to system misconfiguration or resource constraints that must be
DOC> fixed before attempting to recompile these objects.
DOC>#

OBJECTS WITH ERRORS

             10

DOC> The following query reports the number of exceptions caught during
DOC> recompilation. If this number is non-zero, please query the error
DOC> messages in the table UTL_RECOMP_ERRORS to see if any of these errors
DOC> are due to misconfiguration or resource constraints that must be
DOC> fixed before objects can compile successfully.
DOC> Note: Typical compilation errors (due to coding errors) are not
DOC> logged into this table: they go into DBA_ERRORS instead.
DOC>#

ERRORS DURING RECOMPILATION

                     10

Function created.

PL/SQL procedure successfully completed.

Function dropped.

PL/SQL procedure successfully completed.

SQL> SET SERVEROUTPUT ON;
SQL> EXECUTE DBMS_PREUP.INVALID_OBJECTS;
SYS/SYSTEM INVALID OBJECTS
OWNER |OBJECT_NAME

|OBJECT_TYPE

NON SYS/SYSTEM INVALID OBJECTS
OWNER |OBJECT_NAME

|OBJECT_TYPE

CMS DM_CREDIT_DEBIT_TABLES
PROCEDURE
CMS MIG_ADMINISTRATION
PACKAGE BODY
CMS MIG_APPROVAL
PACKAGE BODY
CMS MIG_BENEFIT_PROVIDER
PACKAGE BODY
CMS MIG_CLAIM
PACKAGE BODY
CMS MIG_CUSTOMER_CARE
PACKAGE BODY
CMS MIG_ELIGIBILITY
PACKAGE BODY
CMS MIG_PAYER
PACKAGE BODY
CMS MIG_REGULATOR
PACKAGE BODY
CMS SP_PBMAPPROVALSTRANSFER
PROCEDURE

PL/SQL procedure successfully completed.

SQL> select name from v$datafile;

NAME

E:\APP\cmsdb\ORADATA\CMSQADB\CMSNEWPR\DATAFILE\O1_MF_SYSTEM_FSHGFK3G_.DBF
E:\APP\cmsdb\ORADATA\CMSQADB\CMSNEWPR\DATAFILE\O1_MF_SYSAUX_FSHGFK2Y_.DBF
E:\APP\cmsdb\ORADATA\CMSQADB\CMSNEWPR\DATAFILE\O1_MF_UNDOTBS1_FSHGFJKM_.DBF
E:\APP\cmsdb\ORADATA\CMSQADB\CMSNEWPR\DATAFILE\O1_MF_USERS_FSHGFK3G_.DBF
E:\APP\cmsdb\ORADATA\CMSQADB\TBS_CMS_01.DBF
E:\APP\cmsdb\ORADATA\CMSQADB\TBS_CMS_02.DBF
E:\APP\cmsdb\ORADATA\CMSQADB\TBS_CMS_03.DBF
E:\APP\cmsdb\ORADATA\CMSQADB\TBS_CMS_04.DBF
F:\CMSQADBDATAFILE\TBS_CMS_05.DBF

9 rows selected.

SQL> desc dba_data_files;
Name Null? Type

FILE_NAME VARCHAR2(513)
FILE_ID NUMBER
TABLESPACE_NAME VARCHAR2(30)
BYTES NUMBER
BLOCKS NUMBER
STATUS VARCHAR2(9)
RELATIVE_FNO NUMBER
AUTOEXTENSIBLE VARCHAR2(3)
MAXBYTES NUMBER
MAXBLOCKS NUMBER
INCREMENT_BY NUMBER
USER_BYTES NUMBER
USER_BLOCKS NUMBER
ONLINE_STATUS VARCHAR2(7)
LOST_WRITE_PROTECT VARCHAR2(7)

SQL> select file_name,tablespace_name from dba_data_files where tablespace_name=
‘SYSTEM’;

FILE_NAME

TABLESPACE_NAME

E:\APP\cmsdb\ORADATA\CMSQADB\CMSNEWPR\DATAFILE\O1_MF_SYSTEM_FSHGFK3G_.DBF
SYSTEM

SQL> alter tablespace system add datafile ‘E:\APP\cmsdb\ORADATA\CMSQADB\CMSNEWPR\DATAFILE\SYSTEM_02.DBF’ size 100m autoextend on;

Tablespace altered.

SQL> select file_name,tablespace_name from dba_data_files where tablespace_name=’SYSAUX’;

FILE_NAME

TABLESPACE_NAME

E:\APP\cmsdb\ORADATA\CMSQADB\CMSNEWPR\DATAFILE\O1_MF_SYSAUX_FSHGFK2Y_.DBF
SYSAUX

SQL> alter tablespace sysaux add datafile ‘E:\APP\cmsdb\ORADATA\CMSQADB\CMSNEWPR\DATAFILE\SYSAUX_02.DBF’ size 100m autoextend on;

Tablespace altered.

SQL> @E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade/preupgrade_fixups.sql

SQL> @E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade/preupgrade_fixups.sql
Executing Oracle PRE-Upgrade Fixup Script

Auto-Generated by: Oracle Preupgrade Script
Version: 19.0.0.0.0 Build: 1
Generated on: 2020-11-05 13:13:41

For Source Database: CMSQADB
Source Database Version: 12.2.0.1.0
For Upgrade to Version: 19.0.0.0.0

Preup Preupgrade
Action Issue Is
Number Preupgrade Check Name Remedied Further DBA Action
—— ———————— ———- ——————————–
1. purge_recyclebin YES None.
2. invalid_objects_exist NO Manual fixup recommended.
3. dictionary_stats YES None.
4. tablespaces_info NO Informational only.
Further action is optional.
5. rman_recovery_version NO Informational only.
Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database is not ready
for upgrade. To resolve the outstanding issues, start by reviewing
the preupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.

SQL>

Start database upgrade to 19c

create listener

C:\Users\mazar>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 – Production on Thu Nov 5 23:52:44 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production
Version 19.3.0.0.0

SQL> select name from v$database;

NAME

CMSQADB

SQL> @E:\app\cmsdb\cfgtoollogs\cmsqadb\preupgrade\postupgrade_fixups.sql

Session altered.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

Package created.

No errors.

Package body created.

PL/SQL procedure successfully completed.

No errors.

Package created.

No errors.

Package body created.

No errors.
Executing Oracle POST-Upgrade Fixup Script

Auto-Generated by: Oracle Preupgrade Script
Version: 19.0.0.0.0 Build: 1
Generated on: 2020-11-05 21:24:24

For Source Database: CMSQADB
Source Database Version: 12.2.0.1.0
For Upgrade to Version: 19.0.0.0.0

Preup Preupgrade
Action Issue Is
Number Preupgrade Check Name Remedied Further DBA Action
—— ———————— ———- ——————————–
6. old_time_zones_exist YES None.
7. dir_symlinks YES None.
8. post_dictionary YES None.
9. post_fixed_objects NO Informational only.
Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database upgrade is not
fully complete. To resolve the outstanding issues, start by reviewing
the postupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.

Session altered.

SQL>

SQL> select VERSION,name,open_mode from v$instance,v$database;

VERSION NAME OPEN_MODE

19.0.0.0.0 CMSQADB READ WRITE

SQL> SELECT * FROM v$timezone_file;

FILENAME VERSION CON_ID

timezlrg_32.dat 32 0

SQL> select TZ_VERSION from registry$database;

TZ_VERSION

    32

C:\Windows\system32>lsnrctl status

LSNRCTL for 64-bit Windows: Version 19.0.0.0.0 – Production on 06-NOV-2020 00:0
:14

Copyright (c) 1991, 2019, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=cms-bah-104-165.ace-ins
com)(PORT=1521)))

STATUS of the LISTENER

Alias LISTENER
Version TNSLSNR for 64-bit Windows: Version 19.0.0.0.0 – Prod
ction
Start Date 05-NOV-2020 22:36:04
Uptime 0 days 1 hr. 27 min. 14 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File E:\app\WINDOWS.X64_193000_db_home\network\admin\liste
er.ora
Listener Log File E:\app\oracle\diag\tnslsnr\cms-bah-104-165\listener\a
ert\log.xml
Listening Endpoints Summary…
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cms-bah-104-165.ace-ins.com)(PORT=1
21)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(PIPENAME=\.\pipe\EXTPROC1521ipc)))
Services Summary…
Service “CLRExtProc” has 1 instance(s).
Instance “CLRExtProc”, status UNKNOWN, has 1 handler(s) for this service…
Service “cmsqadb” has 1 instance(s).
Instance “cmsqadb”, status READY, has 1 handler(s) for this service…
Service “cmsqadbXDB” has 1 instance(s).
Instance “cmsqadb”, status READY, has 1 handler(s) for this service…
The command completed successfully

C:\Windows\system32>