SQL table macros 15: parameterized views revisited

In my previous post, I tried to use views and macros together as an equivalent to parameterized views. Unfortunately, my generic macro can’t handle bind variables, which are crucial because in almost all cases they are the parameters! [UPDATE 2025-03-04: the comment below from Iudith Mentzel made me realize that the generic macro works fine, there was a bug in my most recent tests. Sorry about that! I’ll keep this post up as a tribute to Iudith’s comment :-)]

Here is an alternative, which is really just a variant of the solution published by Andrej Pashchenko: see https://blog.sqlora.com/en/parameterized-views-with-sql-macros-part-2/

As before, I’ll concentrate on these advantages of views:

  1. We can include them in queries anywhere a table could appear.
  2. The *_DEPENDENCIES views show the “dependency chain” between them and other objects.

Here is a super simple example:

create or replace view dept_emp as
  select DNAME, LOC, EMPNO, ENAME, JOB
  from dept
  left join emp using(deptno);
select name, type, referenced_name, referenced_type
from user_dependencies where name = 'DEPT_EMP';
NAME        TYPE    REFERENCED_NAME    REFERENCED_TYPE
DEPT_EMP    VIEW    DEPT               TABLE
DEPT_EMP    VIEW    EMP                TABLE

Now, what if we want to filter on DEPTNO? We can just copy the query from the view and add a filter:

create or replace function dept_emp_macro(p_deptno number)
return varchar2 sql_macro is
begin
  return '
select DNAME, LOC, EMPNO, ENAME, JOB
from dept
left join emp using(deptno)
WHERE DEPTNO = P_DEPTNO';
end dept_emp_macro;
/
select * from dept_emp_macro(10);
DNAME      LOC      EMPNO ENAME  JOB      
ACCOUNTING NEW YORK  7839 KING   PRESIDENT
ACCOUNTING NEW YORK  7782 CLARK  MANAGER  
ACCOUNTING NEW YORK  7934 MILLER CLERK    
select name, type, referenced_name, referenced_type
from user_dependencies where name = 'DEPT_EMP_MACRO';
NAME              TYPE        REFERENCED_NAME    REFERENCED_TYPE    
DEPT_EMP_MACRO    FUNCTION    STANDARD           PACKAGE  

With the macro, we have a parameterized query but we have lost the information about the dependency chain in USER_DEPENDENCIES, so the second advantage of views is lost. Not only that, but the first advantage is limited as well: currently SQL macros inside WITH clauses are not permitted, and WITH clauses inside SQL macros cannot use scalar parameters!

My proposal [UPDATE 2025-03-04: see my previous post for my preferred solution]

As with Andrej’s solution, the trick is to write a view that works, but can be changed to inject macro parameters that will act as bind variables. I want the changed code to be as much like the view as possible, so the view uses existing columns as placeholders for the macro parameters and I simply put a “P_” prefix in front. Of course, the names in the view and in the macro have to be coordinated.

Here’s the view. It uses two different approaches to filter the rows of the tables I access. The view works and all the code is tested, even though no rows are filtered yet.

create or replace view DEPT_EMP_PV as
select DNAME, LOC, EMPNO, ENAME, JOB
from (
  select DEPTNO, DNAME, LOC from DEPT
  where NVL(/*<PARAM*/DNAME/*PARAM>*/, DNAME) = DNAME
)
join (
  select DEPTNO, EMPNO, ENAME, JOB
  from EMP E
  where /*<PARAM*/E.JOB/*PARAM>*/ is null
    or E.JOB = /*<PARAM*/E.JOB/*PARAM>*/
)
using(DEPTNO);

In the view, I compare each column to itself. In the macro, I replace each column in a PARAM sandwich by the macro parameter that follows the same naming convention. Note that the “E.” part is ignored.

create or replace function DEPT_EMP_SQM ( -- The function name is specific
  P_VIEW in DBMS_TF.TABLE_T,
  P_DNAME in varchar2 default null,   -- specific parameter
  P_JOB in varchar2 default null      -- specific parameter
) return clob sql_macro is
  L_ROW DEPT_EMP_PV%rowtype;          -- unused but establishes dependency
  L_QUERY long;
begin
  select text into L_QUERY 
  from ALL_VIEWS 
  where OWNER = nvl(trim('"' from P_VIEW.SCHEMA_NAME), user)
    and VIEW_NAME = trim('"' from P_VIEW.TABLE_NAME);
  L_QUERY := REGEXP_REPLACE(
    L_QUERY,
    '\/\*<PARAM\*\/\s*(\S+\.)?(\S+)\s*\/\*PARAM>\*\/',
    'P_\2'
  );
  DBMS_OUTPUT.PUT_LINE(L_QUERY);
  return L_QUERY;
end DEPT_EMP_SQM;
/

Here is the actual text returned by the macro to the Oracle optimizer:

select DNAME, LOC, EMPNO, ENAME, JOB
from (
  select DEPTNO, DNAME, LOC from DEPT
  where NVL(P_DNAME, DNAME) = DNAME
)
join (
  select DEPTNO, EMPNO, ENAME, JOB
  from EMP E
  where P_JOB is null
    or E.JOB = P_JOB
)
using(DEPTNO)

To call the macro: select * from DEPT_EMP_SQM (DEPT_EMP_PV, :dname); for example.

Why do I make you type the name of the view? Because that establishes a dependency between your query and the view, so if the view changes the query will be invalidated and reparsed. I also place a reference to the view inside the macro (see line 6 above), so that the *_DEPENDENCIES views show the relationship between the macro and the view.

Calculating datetime differences

Finding the difference between two dates or timestamps is hard! I am not fully satisfied with any of the solutions I have already proposed (or read), but I think this time I may have a decent answer.

Requirements

I think most people who ask this question:

  1. want the difference between two datetime ( date or timestamp) values, expressed in years, months, days, hours, minutes and seconds;
  2. want the years to be calendar years and the months to be calendar months;
  3. want non-negative integers, except that seconds can have fractions;
  4. Maximum values: for months 11, for days 30, for hours 23, for minutes and seconds 59 (plus the fractional seconds);
  5. I would add a requirement that I call the “round trip”: I should be able to calculate the end date from the start date plus the difference, and I should be able to calculate the start date from the end date minus the difference. What good is a difference if I can’t even confirm that I calculated it correctly?

What tools will I use?

I like to use “interval expressions” for this subject, because they operate with any combination of dates and timestamps, and because their very purpose is to return the difference between two values. Also, they return intervals, which already contain either years and months or days, hours, minutes and seconds. This should be easy then? We’ll see…

Day to second

The day to second interval is pretty easy to understand and use. Days always have 24 hours, which always have 60 minutes, which always have 60 seconds, so the difference we calculate should allow us to do “round trips” with no problem. I’ll start by creating a table with 4 lines of test data.

create table data(start_datetime, end_datetime) as
select date '2023-02-16', timestamp '2023-03-01 12:34:56.7'
from dual
union all
select date '2023-02-16', timestamp '2023-03-30 12:34:56.7'
from dual
union all
select date '2023-02-03', timestamp '2023-03-30 00:00:00'
from dual
union all
select date '2023-02-03', timestamp '2023-04-02 00:00:00'
from dual;
  
alter session set NLS_TIMESTAMP_FORMAT='yyyy-mm-dd hh24:mi:ss.FF1';

select start_datetime, end_datetime,
  (end_datetime - start_datetime) day to second ids
from data;
START_DATETIME END_DATETIME IDS
2023-02-16 00:00:00 2023-03-01 12:34:56.7 +13 12:34:56.7
2023-02-16 00:00:00 2023-03-30 12:34:56.7 +42 12:34:56.7
2023-02-03 00:00:00 2023-03-30 00:00:00.0 +55 00:00:00.
2023-02-03 00:00:00 2023-04-02 00:00:00.0 +58 00:00:00.
 

This result looks promising: we can see the days, minutes, hours and seconds (with fractions!) and we can mix dates and timestamps. Let’s see if we can do round trips:

with diff as (
  select start_datetime, end_datetime,
    (end_datetime - start_datetime) day to second ids
  from data
)
select start_datetime,
  end_datetime - ids start_calculated,
  end_datetime,
  cast(start_datetime as timestamp) + ids end_calculated
from diff
START_DATETIME START_CALCULATED END_DATETIME END_CALCULATED
2023-02-16 00:00:00 2023-02-16 00:00:00.0 2023-03-01 12:34:56.7 2023-03-01 12:34:56.7
2023-02-16 00:00:00 2023-02-16 00:00:00.0 2023-03-30 12:34:56.7 2023-03-30 12:34:56.7
2023-02-03 00:00:00 2023-02-03 00:00:00.0 2023-03-30 00:00:00.0 2023-03-30 00:00:00.0
2023-02-03 00:00:00 2023-02-03 00:00:00.0 2023-04-02 00:00:00.0 2023-04-02 00:00:00.0
 

Bingo! We have a correct and useful datetime difference. All we are missing are the years and months.

Year to month

Year to month intervals have to be handled with care:

  • No fractions: each date is rounded to the nearest month before the difference is evaluated.
  • When added to a datetime, they never change the day of the month: if the result contains an invalid day for that month, you get the nasty “ORA-01839: date not valid for month specified”.

At the very least, we need to combine year to month and day to second intervals to get an acceptable result. With year to month intervals we need to avoid any “rounding”: I always truncate both datetimes to the beginning of the month before calculating the difference. We’ll see how this works right away.

Divide and conquer

Let’s start by separating the components:

  1. Get the dates beginning each month.
  2. Get the year to month interval between the above dates.
  3. Get the day to second intervals
    • between the beginning of the start month and the start date
    • between the beginning of the end month and the end date.
  4. The difference is the year to month interval in 2. plus the difference between the two intervals in 3.
  5. I’ll add at the end the way to calculate each datetime from the other datetime and the difference.
with dates as (  
  select start_datetime, end_datetime,
    trunc(start_datetime,'mm') start_month,
    trunc(end_datetime,'mm') end_month
  from data
)
, intervals as (
  select start_datetime, end_datetime,
    (end_month - start_month) year to month iym,
  (start_datetime - start_month) day to second start_ids,
  (end_datetime - end_month) day to second end_ids
  from dates
)
, answer as (
  select start_datetime, end_datetime, iym,
    end_ids - start_ids ids
  from intervals
)
select end_datetime - ids - iym start_new,
  start_datetime, iym, ids, end_datetime,
  cast(start_datetime as timestamp) + iym + ids end_new
from answer a
START_NEW START_DATETIME IYM IDS END_DATETIME END_NEW
2023-02-16 00:00:00.0 2023-02-16 00:00:00 +00000-01 -14 11:25:03.3 2023-03-01 12:34:56.7 2023-03-01 12:34:56.7
2023-02-16 00:00:00.0 2023-02-16 00:00:00 +00000-01 +14 12:34:56.7 2023-03-30 12:34:56.7 2023-03-30 12:34:56.7
2023-02-03 00:00:00.0 2023-02-03 00:00:00 +00000-01 +27 00:00:00. 2023-03-30 00:00:00.0 2023-03-30 00:00:00.0
2023-02-03 00:00:00.0 2023-02-03 00:00:00 +00000-02 -01 00:00:00. 2023-04-02 00:00:00.0 2023-04-02 00:00:00.0
 

This answer meets all my requirements, except that when start_ids is greater than end_ids, the day to second interval is negative. Saying that the difference between February 16th and March 1st is one month minus 14 days doesn’t make much sense.

Adjust boundaries when necessary

When start_ids is greater than end_ids, something more needs to happen. Having tried various solutions, I now think a reasonable adjustment is as follows:

  1. Calculate the day to second interval from the start date to the beginning of the next month.
    I’ll call this interval start_to_next_ids.
  2. Calculate the year to month interval from the following month to the end month.
    I’ll call this interval next_iym. It will be one month less than what I’ll now call full_iym.
  3. Calculate end_ids as before.
  4. The total difference is start_to_next_ids + next_iym + end_ids.
with dates as (  
  select start_datetime, end_datetime,
    trunc(start_datetime,'mm') start_month,
    trunc(start_datetime,'mm') + interval '1' month start_next_month,
    trunc(end_datetime,'mm') end_month
  from data
)
, intervals as (
  select start_datetime, end_datetime,
    (end_month - start_month) year to month full_iym,
    (end_month - start_next_month) year to month next_iym,
  (start_datetime - start_month) day to second start_ids,
  (start_next_month - start_datetime) day to second start_to_next_ids,
  (end_datetime - end_month) day to second end_ids
  from dates
)
select start_datetime, end_datetime,
  case
    when start_ids > end_ids then start_to_next_ids
    else interval '0' second
  end start_to_next_ids,
  case
    when start_ids > end_ids then next_iym
    else full_iym
  end iym,
  case
    when start_ids > end_ids then end_ids
    else end_ids - start_ids
  end end_ids
from intervals
START_DATETIME END_DATETIME START_TO_NEXT_IDS IYM END_IDS
2023-02-16 00:00:00 2023-03-01 12:34:56.7 +13 00:00:00. +00000-00 +00 12:34:56.7
2023-02-16 00:00:00 2023-03-30 12:34:56.7 +00 00:00:00. +00000-01 +14 12:34:56.7
2023-02-03 00:00:00 2023-03-30 00:00:00.0 +00 00:00:00. +00000-01 +27 00:00:00.
2023-02-03 00:00:00 2023-04-02 00:00:00.0 +26 00:00:00. +00000-01 +01 00:00:00.
 

This answer also allows round trips:

  • end_datetime = start_date_time + start_to_next_ids + iym + end_ids
  • start_datetime = end_date_time - end_ids - iym - start_to_next_ids

But we want one day to second interval!

OK, then let’s just combine the two we have. The with clauses don’t change:

with dates as (  
  select start_datetime, end_datetime,
    trunc(start_datetime,'mm') start_month,
    trunc(start_datetime,'mm') + interval '1' month start_next_month,
    trunc(end_datetime,'mm') end_month
  from data
)
, intervals as (
  select start_datetime, end_datetime,
    (end_month - start_month) year to month full_iym,
    (end_month - start_next_month) year to month next_iym,
  (start_datetime - start_month) day to second start_ids,
  (start_next_month - start_datetime) day to second start_to_next_ids,
  (end_datetime - end_month) day to second end_ids
  from dates
)
select start_datetime, end_datetime,
  case
    when start_ids > end_ids then next_iym
    else full_iym
  end iym,
  case
    when start_ids > end_ids then start_to_next_ids + end_ids
    else end_ids - start_ids
  end ids
from intervals
START_DATETIME END_DATETIME IYM IDS
2023-02-16 00:00:00 2023-03-01 12:34:56.7 +00000-00 +13 12:34:56.7
2023-02-16 00:00:00 2023-03-30 12:34:56.7 +00000-01 +14 12:34:56.7
2023-02-03 00:00:00 2023-03-30 00:00:00.0 +00000-01 +27 00:00:00.
2023-02-03 00:00:00 2023-04-02 00:00:00.0 +00000-01 +27 00:00:00.
 

This is probably the answer most people want. Strangely enough we can do a “round trip” from end_datetime to start_datetime ( I’ll leave that as an exercise for the reader 😉 ), but when going from start_datetime using the same intervals we can occasionally get two different valid end_datetimes, as shown above in rows 3 and 4.

[Update 2025-01-08] Using only the results from the last query, with one combined IDS, we cannot always determine the end datetime from the start datetime and the difference. However, we can do it with one additional piece of information: whether “start_ids” is greater than “end_ids” or not. That is enough for us to use the correct logic, split the IDS into parts if necessary and apply all the intervals in the correct order. I’ll spare you the details…

SQL table macros 14: parameterized views

[UPDATE 2025-03-04: Yesterday I thought this solution did not work with bind variables, but Iudith Mentzel has corrected me, it works fine. My latest tests were buggy. Sorry for the confusion…]

Developers like to reuse code. In SQL, views are reusable queries with lots of advantages, but one drawback: no parameters to limit the queries to only those rows we want. Sure, we can add a WHERE clause when selecting from a view, but the view might return lots of unnecessary data before the WHERE clause filters the rows.

As Andrej Pashchenko recently wrote, SQL table macros can get the results we would want from parameterized views (if they existed), but they do not have all the advantages of real views: see https://blog.sqlora.com/en/parameterized-views-with-sql-macros-part-2/ He wants to combine the advantages of views and macros, and so do I.

(To skip the explanations and see what I came up with, go to My proposal.)

I’ll concentrate on these advantages of views:

  1. We can include them in queries anywhere a table could appear.
  2. The *_DEPENDENCIES views show the “dependency chain” between them and other objects.

Here is a super simple example:

create or replace view dept_emp as
  select deptno, dname, loc, empno, ename, job, mgr, sal, comm
  from dept
  left join emp using(deptno);
select name, type, referenced_name, referenced_type
from user_dependencies where name = 'DEPT_EMP';
NAME        TYPE    REFERENCED_NAME    REFERENCED_TYPE
DEPT_EMP    VIEW    DEPT               TABLE
DEPT_EMP    VIEW    EMP                TABLE

Now, what if we want to filter on DEPTNO? We can just copy the query from the view and add a filter:

create or replace function dept_emp_macro(p_deptno number)
return varchar2 sql_macro is
begin
  return '
select deptno, dname, loc, empno, ename, job, mgr, sal, comm
from dept
left join emp using(deptno)
WHERE DEPTNO = P_DEPTNO';
end dept_emp_macro;
/
select * from dept_emp_macro(10);
DEPTNO DNAME      LOC      EMPNO ENAME  JOB        MGR  SAL COMM
    10 ACCOUNTING NEW YORK  7839 KING   PRESIDENT      5000
    10 ACCOUNTING NEW YORK  7782 CLARK  MANAGER   7839 2450
    10 ACCOUNTING NEW YORK  7934 MILLER CLERK     7782 1300
select name, type, referenced_name, referenced_type
from user_dependencies where name = 'DEPT_EMP_MACRO';
NAME              TYPE        REFERENCED_NAME    REFERENCED_TYPE    
DEPT_EMP_MACRO    FUNCTION    STANDARD           PACKAGE  

With the macro, we have a parameterized query but we have lost the information about the dependency chain in USER_DEPENDENCIES, so the second advantage of views is lost. Not only that, but the first advantage is limited as well: currently SQL macros inside WITH clauses are not permitted, and WITH clauses inside SQL macros cannot use scalar parameters!

My proposal

  1. I write and test a complete query. A WITH clause at the beginning contains hard-coded parameters.
  2. I create a view based on that query, including the hard_coded parameters.
  3. A generic SQL macro returns the view text after stripping off the hard-coded WITH clause.
  4. I call the macro using a new WITH clause, that can include whatever values or bind variables I want.
create or replace function parameterized_view(
  p_view dbms_tf.table_t,
  P_PARMS dbms_tf.table_t
)
return clob sql_macro is
  l_view_text long;
begin
  select text into l_view_text from user_views
  where view_name = trim('"' from p_view.table_name);
  l_view_text :=
    regexp_replace(
      regexp_replace(l_view_text, 'with\s*p_parms[^)]*\)\s*', '', 1, 0, 'i'),
      '^,', 'with '
    );
  return l_view_text;
end parameterized_view;
/
create or replace view dept_emp_pv as
with P_PARMS as (select 'RESEARCH' dname, 'CLERK' job from dual)
, d as (
  select deptno, dname, loc from dept
  where dname = (select dname from P_PARMS)
)
, e as (
  select deptno, empno, ename, job, mgr, sal, comm
  from emp
  where job = (select job from P_PARMS)
)
select * from d left join e using(deptno);
with PARMS as (select 'SALES' dname, 'SALESMAN' job from dual)
select * from parameterized_view(dept_emp_pv, PARMS);
DEPTNO DNAME    LOC        EMPNO    ENAME     JOB         MGR     SAL    COMM
    30 SALES    CHICAGO    7499     ALLEN     SALESMAN   7698    1600     300
    30 SALES    CHICAGO    7521     WARD      SALESMAN   7698    1250     500
    30 SALES    CHICAGO    7654     MARTIN    SALESMAN   7698    1250    1400
    30 SALES    CHICAGO    7844     TURNER    SALESMAN   7698    1500

Note that in the view, the “parameters” clause must have the same name as the second parameter in the macro: P_PARMS. When I call the macro, I can give the clause any name I want, since I pass the name to the macro.

Remember, DEPT_EMP_PV is the identifier of a table or view, so if it is invalidated then the query will automatically be reparsed and the new text will be picked up from the data dictionary. The dependency chain will work even though the macro itself has not been invalidated.

To conclude, using one 16-line generic macro we can have almost all the advantages of views while adding support for parameters! The views do have to be written with this usage in mind, and they are meant to be used as main queries, not subqueries.

SQL table macros 13: compare with primary key

I just updated my COMPARE_ROWSETS function to handle an optional primary or unique key.

If that optional parameter is left out, the function uses the venerable GROUP BY method that handles duplicate rows.

If key columns are indicated, then the “Z#_OP” column can contain updates :

  • U: Updated data (as present in p_new_table)
  • O: Old data (as present in p_old_table)

There is no need for a “Z#_CNT” column since there can be no duplicate rows.

Here is an example of use with a key column :

drop table emp2 purge;
 
create table emp2 as select * from emp;
 
update emp2 set ename = ename||' KONG' where rownum = 1;
 
insert into emp2 
select EMPNO+1000, ENAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO
from emp2 where rownum <= 4 and job != 'PRESIDENT';

select * from compare_rowsets(
  emp, emp2,
  p_key_cols => columns(empno),
  p_exclude_cols => columns(comm)
)
order by empno, z#_op;
Z#_OP EMPNO ENAME JOB MGR HIREDATE SAL DEPTNO
O 7839 KING PRESIDENT   1981-11-17 00:00:00 5000 10
U 7839 KING KONG PRESIDENT   1981-11-17 00:00:00 5000 10
I 8566 JONES MANAGER 7839 1981-04-02 00:00:00 2975 20
I 8698 BLAKE MANAGER 7839 1981-05-01 00:00:00 2850 30
I 8782 CLARK MANAGER 7839 1981-06-09 00:00:00 2450 10
I 8788 SCOTT ANALYST 7566 1987-04-19 00:00:00 3000 20
 

SQL table macros 12: compare tables or queries

The GROUP BY method for comparing table data just turned 18. To celebrate, let’s make it a macro.

My second ever blog post described a method, popularised by Tom Kyte, for comparing the contents of tables or queries. As part of my COMP_SYNC package, the COMPARE_SQL function returns a SELECT statement that uses that method. With a macro, instead of returning the statement I can just execute it and compare tables, views and / or named subqueries.

[UPDATE 2022-02-25 : there is now an optional parameter to indicate a primary or unique key.]

  • I call the two data sources “rowsets”: one is “old”, the other is “new”.
  • Optional excluded columns are excluded from the comparison and the output.
  • Optional key columns must identify all rows uniquely.
    • With key columns, output column “Z#_OP” indicates the type of change:
      • I: Insert into old (present in new, not old)
      • D: Delete from old (present in old, not new)
      • U: Updated data (as present in new)
      • O: Old data (as present in old)
    • Without key columns, the output contains only ‘D’ and ‘I’ rows, and column “Z#_CNT” shows the number of rows to be deleted or inserted.
  • The output contains the values used for the comparison, which may differ from the actual values of the data if the data types do not allow direct comparison (LOBs for example).

The code makes heavy use of my SQM_UTIL package to configure the SELECT template. Depending on the data type of each column, an expression is applied to allow comparisons. For example, LOB column content is replaced by a hash and user-defined types are replaced by JSON.

create or replace function compare_rowsets(
  p_old_table in dbms_tf.table_t,
  p_new_table in dbms_tf.table_t,
  p_key_cols in dbms_tf.columns_t default null,
  p_exclude_cols in dbms_tf.columns_t default null
) return clob sql_macro is
/*
Compares tables, views or named subqueries; one is "old", one is "new".
Optional excluded columns are excluded from the comparison and the output.

The output contains the values used for the comparison, which may differ
from the actual values of the data if the data types do not allow
direct comparison (LOBs for example).

Column "Z#_OP" indicates the type of change:
- I: Insert into old (present in new, not old)
- D: Delete from old (present in old, not new)
- U: Updated data (as present in new)
- O: Old data (as present in old)

If present, key column(s) must identify all rows uniquely.

Without key columns, the output contains only 'D' and 'I' rows,
and column "Z#_CNT" shows the number of rows to be deleted or inserted.
*/
  l_col_column_names_old long;
  l_col_comparables_old long;
  l_col_comparables_new long;
  l_col_keys long;
  l_sql clob;
begin
  sqm_util.col_column_names(p_old_table, l_col_column_names_old, p_exclude_cols);
  sqm_util.col_comparables(p_old_table, l_col_comparables_old, p_exclude_cols);
  sqm_util.col_comparables(p_new_table, l_col_comparables_new, p_exclude_cols);
  
  if p_key_cols is null then
  
    l_sql :=
'select /*+ qb_name(COMPARE) */
  decode(sign(sum(Z#_NEW_CNT)), 1, ''I'', ''D'') Z#_OP,
  abs(sum(Z#_NEW_CNT)) Z#_CNT,
  '|| l_col_column_names_old ||'
FROM (
  select /*+ qb_name(old) */
  '|| l_col_comparables_old ||'
    , -1 Z#_NEW_CNT
  from p_old_table O
  union all
  select /*+ qb_name(new) */
  '|| l_col_comparables_new ||'
    , 1 Z#_NEW_CNT
  from p_new_table N
)
group by
  '|| l_col_column_names_old ||'
having sum(Z#_NEW_CNT) != 0';

  else
    sqm_util.list_columns(p_key_cols, l_col_keys);
    l_sql :=

'select /*+ qb_name(COMPARE) */
  case count(*) over(partition by
    '|| l_col_keys ||'
  ) - Z#_NEW_CNT
    when 0 then ''I''
    when 1 then ''U''
    when 2 then ''D''
    when 3 then ''O''
  end Z#_OP,
  '|| l_col_column_names_old ||'
FROM (
  select
    '|| l_col_column_names_old ||',
    sum(Z#_NEW_CNT) Z#_NEW_CNT
  FROM (
    select /*+ qb_name(old) */
    '|| l_col_comparables_old ||',
    -1 Z#_NEW_CNT
    from p_old_table O
    union all
    select /*+ qb_name(new) */
    '|| l_col_comparables_new ||',
    1 Z#_NEW_CNT
    from p_new_table N
  )
  group by
    '|| l_col_column_names_old ||'
  having sum(Z#_NEW_CNT) != 0
)';

  end if;
  --dbms_output.put_line(l_sql);
  return l_sql;
end compare_rowsets;
/
drop table emp2 purge;
 
create table emp2 as select * from emp;
 
update emp2 set ename = ename||' KONG' where rownum = 1;
 
insert into emp2 
select EMPNO+1000, ENAME, JOB, MGR, HIREDATE, SAL, COMM, DEPTNO
from emp2 where rownum <= 4 and job != 'PRESIDENT';
 
select * from compare_rowsets(
  emp, emp2, 
  p_exclude_cols => columns(comm)
)
order by empno, z#_op;
Z#_OP Z#_CNT EMPNO ENAME JOB MGR HIREDATE SAL DEPTNO
D 1 7839 KING PRESIDENT   1981-11-17 00:00:00 5000 10
I 1 7839 KING KONG PRESIDENT   1981-11-17 00:00:00 5000 10
I 1 8566 JONES MANAGER 7839 1981-04-02 00:00:00 2975 20
I 1 8698 BLAKE MANAGER 7839 1981-05-01 00:00:00 2850 30
I 1 8782 CLARK MANAGER 7839 1981-06-09 00:00:00 2450 10
I 1 8788 SCOTT ANALYST 7566 1987-04-19 00:00:00 3000 20
 

Hierarchical JSON

At last, a use case where PL/SQL is faster than SQL : avoiding unnecessary ordering !

In the last few years, different solutions have been found to represent hierarchical data in JSON format :

I won’t go into the SQL solutions in detail. Basically there are three stages :

  1. Get the data in hierarchical order and return the order, the level, and the data as a JSON object.
  2. Using the LEAD() and LAG() functions, determine whether the current JSON object is a child, a parent or a sibling and apply the appropriate JSON “glue”.
  3. Aggregate the result into a CLOB, preserving the original order.

Notice that stages 2 and 3 require the data to be in the order established by stage 1. In the SQL solutions, the execution plan shows that unnecessary sorts were done, sometimes requiring the use of TEMP space : yuck !

I finally realised that a PL/SQL function could read the output from stage 1, and for each input row add the JSON “glue” and append the result to the CLOB that the function returns. No extra sorting, and it runs about 40% faster than the most optimal SQL solution. Let me explain a few details :

  • The main input is a CURSOR expression : this is not a literal, but a real live SELECT statement that can include bind variables. It returns only the level and a JSON object with the desired data, so it is quite generic. The ORDER SIBLINGS BY clause is optional.
  • The input parameter is a strongly typed REF CURSOR, which allows it to be fetched into a table of P/SQL records.
  • The input is fetched using BULK COLLECT with LIMIT, which uses little memory. Special handling is needed, because I need to know the levels of the rows immediately preceding and following the row being processed.
  • If a JSON object is a parent, I add a name / value pair where the value is an array. The name is by default ‘children’ but it is configurable by an optional second parameter.
  • Concatenating text to a CLOB is a costly operation. To speed it up, I fill up a VARCHAR2 buffer, then concatenate the buffer to the CLOB.
  • Use PLSQL_OPTIMIZE_LEVEL=3 in order to “inline” the calls to APPEND_JSO. This shaves off about .2 seconds per million rows processed.
create or replace package hier_to_clob is
  type t_hier_rec is record(
    lvl pls_integer,
    jso varchar2(4000)
  );
  type tt_hier_rec is table of t_hier_rec;
  TYPE t_hier_rc IS REF CURSOR RETURN t_hier_rec;
  function get(
    p_hier_rc in t_hier_rc,
    p_array_name varchar2 default 'children',
    p_limit integer default 500
  ) return clob;
end hier_to_clob;
/
create or replace package body hier_to_clob is
  function get(
    p_hier_rc in t_hier_rc,
    p_array_name varchar2 default 'children',
    p_limit integer default 500
  ) return clob is
    l_array_name varchar2(130) := ',"'||p_array_name||'":[';
    lt_hier_rec tt_hier_rec;
    l_hier_rec_prev t_hier_rec := t_hier_rec(0, null);
    l_lvl_prev2 pls_integer := 0;
    l_clob clob;
    l_buffer varchar2(32767) := null;
    l_buflen pls_integer := 0;
    do_prev boolean:= false;
    procedure append_jso(
      p_jso varchar2,
      p_lvl_prev pls_integer,
      p_lvl pls_integer,
      p_lvl_next pls_integer
    ) is
      l_jso varchar2(4000);
    begin
      l_jso :=
        case
          when p_lvl_prev = 0 then null
          when p_lvl - p_lvl_prev = 1 then l_array_name
          when p_lvl > 1 then ','
        end ||
        rtrim(p_jso, '}') ||
        rpad('}', (p_lvl - p_lvl_next) * 2 + 1, ']}');

      if l_buflen + lengthb(l_jso) > 32767 then
        l_clob := l_clob || l_buffer;
        l_buffer := l_jso;
        l_buflen := lengthb(l_buffer);
      else
        l_buffer := l_buffer || l_jso;
        l_buflen := l_buflen + lengthb(l_jso);
      end if;
    end append_jso;
  begin
    loop
      fetch p_hier_rc bulk collect into lt_hier_rec limit p_limit;
      if do_prev then
        append_jso(
          l_hier_rec_prev.jso,
          l_lvl_prev2,
          l_hier_rec_prev.lvl,
          case when lt_hier_rec.count > 0 then lt_hier_rec(1).lvl else 1 end
        );
        do_prev := false;
      end if;
      for i in 1..lt_hier_rec.count-1 loop
        append_jso(
          lt_hier_rec(i).jso,
          case
            when i = 1 then l_hier_rec_prev.lvl
            else lt_hier_rec(i-1).lvl
          end,
          lt_hier_rec(i).lvl,
          lt_hier_rec(i+1).lvl
        );
      end loop;
      if lt_hier_rec.count > 0 then
        l_lvl_prev2 :=
          case lt_hier_rec.count
            when 1 then l_hier_rec_prev.lvl
            else lt_hier_rec(lt_hier_rec.count-1).lvl
          end;
        l_hier_rec_prev := lt_hier_rec(lt_hier_rec.count);
        do_prev := true;
      end if;
      exit when p_hier_rc%notfound;
    end loop;
    if do_prev then
      append_jso(
        l_hier_rec_prev.jso,
        l_lvl_prev2,
        l_hier_rec_prev.lvl,
        1
      );
    end if;
    if l_buflen > 0 then
      l_clob := l_clob || l_buffer;
    end if;
    return l_clob;
  end get;
end hier_to_clob;
/
select hier_to_clob.get(
  cursor(
    select level, json_object(empno, ename)
    from emp
    start with empno = 7566
    connect by mgr = prior empno
  ),
  'grunts'
)
from dual;
/* pretty printing done outside of the function */
{
  "empno" : 7566,
  "ename" : "JONES",
  "grunts" :
  [
    {
      "empno" : 7788,
      "ename" : "SCOTT",
      "grunts" :
      [
        {
          "empno" : 7876,
          "ename" : "ADAMS"
        }
      ]
    },
    {
      "empno" : 7902,
      "ename" : "FORD",
      "grunts" :
      [
        {
          "empno" : 7369,
          "ename" : "SMITH"
        }
      ]
    }
  ]
}

SQL table macros 11: JUST_PIVOT for 21c (and 19c)

[HUGE UPDATE 2022-01-14 15:45 GMT: thanks to Iudith Mentzel, here is a much simpler and better solution than the one I originally posted today!]

[UPDATE 2022-01-15: to ensure that the INVALIDATE_OBJECT function is called only once, I made it DETERMINISTIC; I also wrapped the call in a scalar subquery, as Iudith Mentzel suggested in her comment below.]

My previous JUST_PIVOT solution for dynamic pivoting does not work in Oracle DB 21c: sorry about that!

Here’s the explanation:

  1. When we want to pivot rows to columns, the columns may change if the data changes.
  2. A SQL macro can change the columns, but only during the hard parse phase.
  3. My previous attempt forces a hard parse every time by using a “flashback query”:
    • All parsed statements cause a “cursor” to be placed in the shared pool;
    • cursors that can be “shared” are reused if the same statement is submitted again;
    • cursors from flashback queries are marked “unshareable”, which means the cursor cannot be be reused, so there is a hard parse at every execution.
  4. Starting with version 21c, flashback query cursors can be shared, which means that technique no longer works.

Is this a bug or a feature?

Why, after all these years, have flashback queries become shareable? Was this really a conscious decision by Oracle, or is it a bug that Oracle will “fix” and Make my Macro Great Again? Thanks to a tweet by Martin Berger (@martinberx) which got me to read this article by Tim Hall , I confirmed that this new behavior is intended:

select description from V$SYSTEM_FIX_CONTROL
where bugno = 10123661;

DESCRIPTION
--------------
Enable cursor sharing for AS OF queries
 

Alternative solution

This is based on a brilliant suggestion by Iudith Mentzel on Twitter (@mentzel_iudith):

  • Create a function that invalidates the SQL macro using DBMS_UTILITY.INVALIDATE;
  • Have the generated SQL statement call that function.
    • The macro will have finished, so the invalidation does not interfere with the macro execution;
    • The invalidation does not commit, so no bad side effects.
    • Also no child cursors! That was the drawback of the AS OF technique.
create or replace function invalidate_object (
  p_owner in varchar2,
  p_object_name in varchar2
) return number deterministic is
  l_object_id number;
begin
  select object_id into l_object_id
  from all_objects
  where (owner, object_name) = ((p_owner, p_object_name));
  dbms_utility.invalidate(
    p_object_id => l_object_id,
    p_option_flags => dbms_utility.INV_ERROR_ON_RESTRICTIONS
  );
  return 0;
end invalidate_object;
/

create or replace function just_pivot(
  p_INPUT_QUERY in sys.odcivarchar2list,
  p_GROUP_COLS in dbms_tf.columns_t,
  p_PIVOT_COLS in dbms_tf.columns_t,
  p_AGG_COLS in dbms_tf.columns_t,
  p_total_label in dbms_tf.columns_t default null   
) return clob sql_macro is

/* Just Pivot MACRO
This macro takes as input a query that does an explicit GROUP BY.
The macro first uses the input query to get the list of pivot columns,
then wraps the input query in a final query that does a PIVOT using MAX().

Input:
1) p_INPUT_QUERY: the input query must be a literal
   wrapped in a sys.odcivarchar2list() collection.
   This allows the content to be accessed by the macro
   (macros do not have access to the content of VARCHAR2 literals).

Every column in the input query must also appear in one of these column lists:
2) p_GROUP_COLS: columns that are GROUPed BY explicitly in the input query
   and implicitly in the PIVOT
3) p_PIVOT_COLS: column that are GROUPed BY explicitly in the input query
   and pivoted in the PIVOT
4) p_AGG_COLS: column that are aggregated explicitly in the input query
   and "re-aggregated" using MAX() in the PIVOT
5) p_total_label: label to use for pivot totals (if CUBE, ROLLUP, etc.)
   null values are assumed to be totals and will be replaced with the label
   ('Total' is the default) and will be ordered at the end

Processing:
- within the macro, execute the input query to get the list of pivot columns
- generate the final SQL statement using input query and intermediate result
- the final SQL statement will call a function to invalidate this function,
  which will invalidate the cursor and cause the query to be parsed
  at every execution.
*/

/*
- PIVOT_IN_LIST concatenates values from all p_PIVOT_COLS columns
  for example, if (DNAME,JOB) then
    ('ACCOUNTING','CLERK') as "ACCOUNTING_CLERK",
    ('ACCOUNTING','MANAGER') as "ACCOUNTING_MANAGER",
  and so on
*/
  l_pivot_in_list_sql long :=  
q'@select listagg('(' || EXPR || ') as "' || AL || '"', ',
') within group(order by #PIVOT_COLS#)
from (
  select distinct
  #EXPR# EXPR,
  #ALIAS# AL,
  #PIVOT_COLS#
  from (#INPUT_QUERY#)
)@';
  l_PIVOT_COLS varchar2(4000);
  l_EXPR varchar2(4000);
  l_ALIAS varchar2(4000);

  l_final_sql long :=
'select * from (
  select #NVL_COLS#,
  #AGG_COLS#
  from (#INPUT_QUERY#)
  where 0 = (
    select invalidate_object(
      '''||$$PLSQL_UNIT_OWNER||''','''||$$PLSQL_UNIT||'''
    ) from dual
  )
)
pivot(#AGG_MAX_COLS# for (#PIVOT_COLS#) in (
  #PIVOT_IN_LIST#
))
order by #ORDER_COLS#';
  l_NVL_COLS varchar2(4000);
  l_AGG_COLS varchar2(4000);
  l_AGG_MAX_COLS varchar2(4000);
  l_PIVOT_IN_LIST varchar2(4000);
  l_ORDER_COLS varchar2(4000);
  
  l_total_label varchar2(32) := 'Total';

begin

  -- set value of l_total_label, which is needed right away
  if p_total_label is not null then
    sqm_util.list_columns(p_total_label, l_total_label, '%s', null, true);
  end if;
  
  -- set values to be plugged into l_pivot_in_list_sql
  sqm_util.list_columns(p_PIVOT_COLS, l_PIVOT_COLS);
  sqm_util.list_columns(
    p_PIVOT_COLS, l_EXPR,
    '''''''''||nvl(%s||null,'''||l_total_label||''')||''''''''', 
    q'§||','||§',
    true
  );
  sqm_util.list_columns(p_PIVOT_COLS, l_ALIAS,
    'nvl(%s||null,'''||l_total_label||''')',
    q'§||'_'||§',
    true
  );
  
  -- plug values into l_pivot_in_list_sql
  l_pivot_in_list_sql := replace(replace(replace(replace(
    l_pivot_in_list_sql,
    '#PIVOT_COLS#', l_PIVOT_COLS),
    '#EXPR#', l_EXPR),
    '#ALIAS#', l_ALIAS),
    '#INPUT_QUERY#', p_INPUT_QUERY(1)
  );  
  -- dbms_output.put_line(l_pivot_in_list_sql);
  
  -- execute l_pivot_in_list_sql
  execute immediate l_pivot_in_list_sql into l_PIVOT_IN_LIST;
  -- dbms_output.put_line(l_PIVOT_IN_LIST);
  
  -- set values to be plugged into l_final_sql
  sqm_util.list_columns(
    p_GROUP_COLS multiset union p_PIVOT_COLS,
    l_NVL_COLS,
    'nvl(%s||null, '''||l_total_label||''') %s'
  );

  sqm_util.list_columns(p_AGG_COLS, l_AGG_MAX_COLS, 'max(%s) %s');
  sqm_util.list_columns(p_AGG_COLS, l_AGG_COLS);
  sqm_util.list_columns(p_GROUP_COLS, l_ORDER_COLS,
    'nullif(%s, '''||l_total_label||''')'
  );
  
  -- plug values into l_final_sql
  l_final_sql := replace(replace(replace(replace(replace(replace(replace(
    l_final_sql,
    '#NVL_COLS#', l_NVL_COLS),
    '#AGG_COLS#', l_AGG_COLS),
    '#INPUT_QUERY#', p_INPUT_QUERY(1)),
    '#AGG_MAX_COLS#', l_AGG_MAX_COLS),
    '#PIVOT_COLS#', l_PIVOT_COLS),
    '#ORDER_COLS#', l_ORDER_COLS),
    '#PIVOT_IN_LIST#', l_PIVOT_IN_LIST);
  
  dbms_output.put_line(l_final_sql);
  return l_final_sql;
  
end just_pivot;
/

SQL table macros 10: JUST_PIVOT examples

To show the power of the JUST_PIVOT macro, here are several examples. Please refer to:

The examples are based on a minimal dataset which is completely abstract. My apologies for not providing more pleasant data…

drop table t purge;
create table t as
with Col_A as (select 'Val_A'||level Col_A from dual connect by level <= 2),
Col_B as (select 'Val_B'||level Col_B from dual connect by level <= 2),
Col_C as (select 'Val_C'||level Col_C from dual connect by level <= 2),
Col_D as (select 'Val_D'||level Col_D from dual connect by level <= 2)
select * from Col_A,Col_B,Col_C,Col_D;

select * from t order by 1,2,3,4
COL_A COL_B COL_C COL_D
Val_A1 Val_B1 Val_C1 Val_D1
Val_A1 Val_B1 Val_C1 Val_D2
Val_A1 Val_B1 Val_C2 Val_D1
Val_A1 Val_B1 Val_C2 Val_D2
Val_A1 Val_B2 Val_C1 Val_D1
Val_A1 Val_B2 Val_C1 Val_D2
Val_A1 Val_B2 Val_C2 Val_D1
Val_A1 Val_B2 Val_C2 Val_D2
Val_A2 Val_B1 Val_C1 Val_D1
Val_A2 Val_B1 Val_C1 Val_D2
Val_A2 Val_B1 Val_C2 Val_D1
Val_A2 Val_B1 Val_C2 Val_D2
Val_A2 Val_B2 Val_C1 Val_D1
Val_A2 Val_B2 Val_C1 Val_D2
Val_A2 Val_B2 Val_C2 Val_D1
Val_A2 Val_B2 Val_C2 Val_D2
 

One group column, one pivot column, one aggregation column

This simplest example shows that generated column names are based on the value of the pivot column followed by the name of the aggregated column. The PIVOT clause does this for us.

select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select Col_A, Col_B, count(*) cnt
from t
group by Col_A, Col_B
§'
  ),
  p_group_cols => columns(Col_A),
  p_pivot_cols => columns(Col_B),
  p_agg_cols => columns(cnt)
);
Final query generated by the macro
with pivoted as (
  select * from (
    select nvl("COL_A"||null, 'Total') "COL_A",nvl("COL_B"||null, 'Total') "COL_B",
    "CNT"
    from (
select Col_A, Col_B, count(*) cnt
from t
group by Col_A, Col_B
)
  )
  pivot(max("CNT") "CNT" for ("COL_B") in (
    ('Val_B1') as "Val_B1",
('Val_B2') as "Val_B2"
  ))
)
select * from pivoted
as of scn dbms_flashback.get_system_change_number
order by nullif("COL_A", 'Total')

COL_A Val_B1_CNT Val_B2_CNT
Val_A1 4 4
Val_A2 4 4
 

Changing one line of code gives us totals by the group column, by the pivot column and a grand total. There is actually a fifth parameter called p_total_label that you can use to change the ‘Total’ label, but whatever the label is, the same one is used everywhere.

select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select Col_A, Col_B, count(*) cnt
from t
group by cube(Col_A, Col_B)
§'
  ),
  p_group_cols => columns(Col_A),
  p_pivot_cols => columns(Col_B),
  p_agg_cols => columns(cnt)
);
Final query generated by the macro
with pivoted as (
  select * from (
    select nvl("COL_A"||null, 'Total') "COL_A",nvl("COL_B"||null, 'Total') "COL_B",
    "CNT"
    from (
select Col_A, Col_B, count(*) cnt
from t
group by cube(Col_A, Col_B)
)
  )
  pivot(max("CNT") "CNT" for ("COL_B") in (
    ('Val_B1') as "Val_B1",
('Val_B2') as "Val_B2",
('Total') as "Total"
  ))
)
select * from pivoted
as of scn dbms_flashback.get_system_change_number
order by nullif("COL_A", 'Total')

COL_A Val_B1_CNT Val_B2_CNT Total_CNT
Val_A1 4 4 8
Val_A2 4 4 8
Total 8 8 16
 

This example shows how the GROUPING_ID function tells us the GROUP BY level of every “cell” in our output. I’ll use it to eliminate unwanted rows or columns.

select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select Col_A, Col_B, grouping_id(Col_A, Col_B) gid
from t
group by cube(Col_A, Col_B)
§'
  ),
  p_group_cols => columns(Col_A),
  p_pivot_cols => columns(Col_B),
  p_agg_cols => columns(gid)
);
COL_A Val_B1_GID Val_B2_GID Total_GID
Val_A1 0 0 1
Val_A2 0 0 1
Total 2 2 3
 

Let’s say I don’t want the Total_CNT column: I simply choose the GROUPING_ID values I want to keep.

select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select Col_A, Col_B, count(*) cnt
from t
group by cube(Col_A, Col_B)
having grouping_id(Col_A, Col_B) in (0,2)
§'
  ),
  p_group_cols => columns(Col_A),
  p_pivot_cols => columns(Col_B),
  p_agg_cols => columns(cnt)
);
COL_A Val_B1_CNT Val_B2_CNT
Val_A1 4 4
Val_A2 4 4
Total 8 8
 

Two aggregations

The PIVOT clause allows us to aggregate as many values as we want, of different datatypes if we want.

select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select Col_A, Col_B, count(*) cnt, 'Count='||count(*) lit
from t
group by cube(Col_A, Col_B)
§'
  ),
  p_group_cols => columns(Col_A),
  p_pivot_cols => columns(Col_B),
  p_agg_cols => columns(cnt,lit)
);
Final query generated by the macro
with pivoted as (
  select * from (
    select nvl("COL_A"||null, 'Total') "COL_A",nvl("COL_B"||null, 'Total') "COL_B",
    "CNT","LIT"
    from (
select Col_A, Col_B, count(*) cnt, 'Count='||count(*) lit
from t
group by cube(Col_A, Col_B)
)
  )
  pivot(max("CNT") "CNT",max("LIT") "LIT" for ("COL_B") in (
    ('Val_B1') as "Val_B1",
('Val_B2') as "Val_B2",
('Total') as "Total"
  ))
)
select * from pivoted
as of scn dbms_flashback.get_system_change_number
order by nullif("COL_A", 'Total')

COL_A Val_B1_CNT Val_B1_LIT Val_B2_CNT Val_B2_LIT Total_CNT Total_LIT
Val_A1 4 Count=4 4 Count=4 8 Count=8
Val_A2 4 Count=4 4 Count=4 8 Count=8
Total 8 Count=8 8 Count=8 16 Count=16
 

Two group-by columns

With two group-by columns, we would normally show subtotals by both columns, totals by the first column, and the grand total. To avoid getting totals by the second column, we can use the GROUPING_ID function just on those two columns.

select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select Col_A, Col_B, Col_C, count(*) cnt
from t
group by cube(Col_A, Col_B, Col_C)
having grouping_id(Col_A, Col_B) != 2
§'
  ),
  p_group_cols => columns(Col_A, Col_B),
  p_pivot_cols => columns(Col_C),
  p_agg_cols => columns(cnt)
);
Final query generated by the macro
with pivoted as (
  select * from (
    select nvl("COL_A"||null, 'Total') "COL_A",nvl("COL_B"||null, 'Total') "COL_B",nvl("COL_C"||null, 'Total') "COL_C",
    "CNT"
    from (
select Col_A, Col_B, Col_C, count(*) cnt
from t
group by cube(Col_A, Col_B, Col_C)
having grouping_id(Col_A, Col_B) != 2
)
  )
  pivot(max("CNT") "CNT" for ("COL_C") in (
    ('Val_C1') as "Val_C1",
('Val_C2') as "Val_C2",
('Total') as "Total"
  ))
)
select * from pivoted
as of scn dbms_flashback.get_system_change_number
order by nullif("COL_A", 'Total'),nullif("COL_B", 'Total')

COL_A COL_B Val_C1_CNT Val_C2_CNT Total_CNT
Val_A1 Val_B1 2 2 4
Val_A1 Val_B2 2 2 4
Val_A1 Total 4 4 8
Val_A2 Val_B1 2 2 4
Val_A2 Val_B2 2 2 4
Val_A2 Total 4 4 8
Total Total 8 8 16
 

Two pivot columns

Again I’ll show subtotals by both columns, totals by the first column, and the grand total. To avoid getting totals by the second column, use the GROUPING_ID function on the two pivot columns.

select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select Col_A, Col_B, Col_C, count(*) cnt
from t
group by cube(Col_A, Col_B, Col_C)
having grouping_id(Col_B, Col_C) != 2
§'
  ),
  p_group_cols => columns(Col_A),
  p_pivot_cols => columns(Col_B, Col_C),
  p_agg_cols => columns(cnt),
  p_total_label => columns("Tot")
);
Final query generated by the macro
with pivoted as (
  select * from (
    select nvl("COL_A"||null, 'Tot') "COL_A",nvl("COL_B"||null, 'Tot') "COL_B",nvl("COL_C"||null, 'Tot') "COL_C",
    "CNT"
    from (
select Col_A, Col_B, Col_C, count(*) cnt
from t
group by cube(Col_A, Col_B, Col_C)
having grouping_id(Col_B, Col_C) != 2
)
  )
  pivot(max("CNT") "CNT" for ("COL_B","COL_C") in (
    ('Val_B1','Val_C1') as "Val_B1_Val_C1",
('Val_B1','Val_C2') as "Val_B1_Val_C2",
('Val_B1','Tot') as "Val_B1_Tot",
('Val_B2','Val_C1') as "Val_B2_Val_C1",
('Val_B2','Val_C2') as "Val_B2_Val_C2",
('Val_B2','Tot') as "Val_B2_Tot",
('Tot','Tot') as "Tot_Tot"
  ))
)
select * from pivoted
as of scn dbms_flashback.get_system_change_number
order by nullif("COL_A", 'Tot')

COL_A Val_B1_Val_C1_CNT Val_B1_Val_C2_CNT Val_B1_Tot_CNT Val_B2_Val_C1_CNT Val_B2_Val_C2_CNT Val_B2_Tot_CNT Tot_Tot_CNT
Val_A1 2 2 4 2 2 4 8
Val_A2 2 2 4 2 2 4 8
Tot 4 4 8 4 4 8 16
 

Two group-by columns and two pivot columns

Again I’ll show subtotals, totals and the grand total. To avoid getting totals by the second column, use the GROUPING_ID function on the two group-by columns and the two pivot columns.

select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select Col_A, Col_B, Col_C, Col_D, count(*) cnt
from t
group by cube(Col_A, Col_B, Col_C, Col_D)
having 2 not in (
  grouping_id(Col_A, Col_B),
  grouping_id(Col_C, Col_D)
)
§'
  ),
  p_group_cols => columns(Col_A, Col_B),
  p_pivot_cols => columns(Col_C, Col_D),
  p_agg_cols => columns(cnt),
  p_total_label => columns("Tot")
);
Final query generated by the macro
with pivoted as (
  select * from (
    select nvl("COL_A"||null, 'Tot') "COL_A",nvl("COL_B"||null, 'Tot') "COL_B",nvl("COL_C"||null, 'Tot') "COL_C",nvl("COL_D"||null, 'Tot') "COL_D",
    "CNT"
    from (
select Col_A, Col_B, Col_C, Col_D, count(*) cnt
from t
group by cube(Col_A, Col_B, Col_C, Col_D)
having 2 not in (
  grouping_id(Col_A, Col_B),
  grouping_id(Col_C, Col_D)
)
)
  )
  pivot(max("CNT") "CNT" for ("COL_C","COL_D") in (
    ('Val_C1','Val_D1') as "Val_C1_Val_D1",
('Val_C1','Val_D2') as "Val_C1_Val_D2",
('Val_C1','Tot') as "Val_C1_Tot",
('Val_C2','Val_D1') as "Val_C2_Val_D1",
('Val_C2','Val_D2') as "Val_C2_Val_D2",
('Val_C2','Tot') as "Val_C2_Tot",
('Tot','Tot') as "Tot_Tot"
  ))
)
select * from pivoted
as of scn dbms_flashback.get_system_change_number
order by nullif("COL_A", 'Tot'),nullif("COL_B", 'Tot')

COL_A COL_B Val_C1_Val_D1_CNT Val_C1_Val_D2_CNT Val_C1_Tot_CNT Val_C2_Val_D1_CNT Val_C2_Val_D2_CNT Val_C2_Tot_CNT Tot_Tot_CNT
Val_A1 Val_B1 1 1 2 1 1 2 4
Val_A1 Val_B2 1 1 2 1 1 2 4
Val_A1 Tot 2 2 4 2 2 4 8
Val_A2 Val_B1 1 1 2 1 1 2 4
Val_A2 Val_B2 1 1 2 1 1 2 4
Val_A2 Tot 2 2 4 2 2 4 8
Tot Tot 4 4 8 4 4 8 16
 

This function allows any combination of one or more group-by columns, pivot columns and aggregations, limited only by screen real estate and how much horizontal scrolling you want to inflict on the users. In my next post I’ll talk about some limitations.

SQL table macros 9: just pivot!

The PIVOT clause does some aggregation, then pivots the result – but it is not dynamic.

Let’s make a deal: you do the aggregation and the JUST_PIVOT table macro will pivot the result, dynamically!

[UPDATE 2022-01-14: the solution I propose here works only in version 19c. For a more complicated solution that works in 19c and 21c, see SQL table macros 11: JUST_PIVOT for 21c (and 19c)]

Doing it the old-fashioned way

Let’s say we want the total salary for each job type in each department. This is a simple aggregation:

select dname, job, sum(sal) sal
from dept
join emp using(deptno)
group by dname, job
order by 1,2
DNAME JOB SAL
ACCOUNTING CLERK 1300
ACCOUNTING MANAGER 2450
ACCOUNTING PRESIDENT 5000
RESEARCH ANALYST 6000
RESEARCH CLERK 1900
RESEARCH MANAGER 2975
SALES CLERK 950
SALES MANAGER 2850
SALES SALESMAN 5600
 

Now that we know all possible JOB values, we can aggregate and pivot the result directly.

select * from (
  select dname, job, sal
  from dept
  join emp using(deptno)
)
pivot(sum(SAL) SAL for (JOB) in (
  ('ANALYST') as ANALYST,
  ('CLERK') as CLERK,
  ('MANAGER') as MANAGER,
  ('PRESIDENT') as PRESIDENT,
  ('SALESMAN') as SALESMAN
));
DNAME ANALYST_SAL CLERK_SAL MANAGER_SAL PRESIDENT_SAL SALESMAN_SAL
ACCOUNTING   1300 2450 5000  
RESEARCH 6000 1900 2975    
SALES   950 2850   5600
 
  • I call SAL an “aggregation column” because it is used in an aggregation function.
  • JOB is a “pivot column” because its values are used in create the new pivoted columns.
  • DNAME is a “group by column” because PIVOT groups by it implicitly.

Using the JUST_PIVOT macro

JUST_PIVOT asks you for four things:

  1. p_INPUT_QUERY: the input query must be a literal wrapped in a sys.odcivarchar2list() collection. This allows the content to be accessed by the macro; macros do not have access to the content of VARCHAR2 literals.

    Every column in the input query must also appear in one of these column lists:
  2. p_GROUP_COLS: columns that are GROUPed BY explicitly in the input query and implicitly in the PIVOT
  3. p_PIVOT_COLS: columns that are GROUPed BY explicitly in the input query and pivoted in the PIVOT
  4. p_AGG_COLS: column that are aggregated explicitly in the input query and “re-aggregated” using MAX() in the PIVOT
select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select dname, job, sum(sal) sal
from dept
join emp using(deptno)
group by dname, job
§'
  ),
  p_group_cols => columns(dname),
  p_pivot_cols => columns(job),
  p_agg_cols => columns(sal)
);
DNAME ANALYST_SAL CLERK_SAL MANAGER_SAL PRESIDENT_SAL SALESMAN_SAL
ACCOUNTING   1300 2450 5000  
RESEARCH 6000 1900 2975    
SALES   950 2850   5600
 

Advantages of the macro

  • The macro actually executes the input query a first time in order to build a complete “pivot-in-list”. We don’t have to.
  • The final query is a flashback query, so it is parsed every time and the macro generates an up-to-date final query every time.
  • If we generate total values, they are automatically put in the right places in the output!
    Here is an example, with one line of code changed :
select * from just_pivot(
  p_input_query => sys.odcivarchar2list(
q'§
select dname, job, sum(sal) sal
from dept
join emp using(deptno)
group by cube(dname, job)
§'
  ),
  p_group_cols => columns(dname),
  p_pivot_cols => columns(job),
  p_agg_cols => columns(sal)
);
DNAME ANALYST_SAL CLERK_SAL MANAGER_SAL PRESIDENT_SAL SALESMAN_SAL Total_SAL
ACCOUNTING   1300 2450 5000   8750
RESEARCH 6000 1900 2975     10875
SALES   950 2850   5600 9400
Total 6000 4150 8275 5000 5600 29025
 

My next post will show how the macro works with multiple group by, pivot and / or aggregation columns.

The code

/* Just Pivot MACRO
This macro takes as input a query that does an explicit GROUP BY.
The macro first uses the input query to get the list of pivot columns,
then wraps the input query in a final query that does a PIVOT using MAX().
The final query has a flashback clause in order to make the cursor unshareable:
  - there will be a hard parse phase at every execution,
  - the macro will execute and the list of pivot values will be up to date.

Input:
1) p_INPUT_QUERY: the input query must be a literal
   wrapped in a sys.odcivarchar2list() collection.
   This allows the content to be accessed by the macro;
   macros do not have access to the content of VARCHAR2 literals.

Every column in the input query must also appear in one of these column lists:
2) p_GROUP_COLS: columns that are GROUPed BY explicitly in the input query
   and implicitly in the PIVOT (*)
3) p_PIVOT_COLS: column that are GROUPed BY explicitly in the input query
   and pivoted in the PIVOT (*)
4) p_AGG_COLS: column that are aggregated explicitly in the input query
   and "re-aggregated" using MAX() in the PIVOT
5) p_total_label: label to use for pivot totals (if CUBE, ROLLUP, etc.)
   null values are assumed to be totals and will be replaced with some non-null label
   such as 'Total', which is the default, and will be ordered at the end

Processing:
- within the macro, execute the input query to get the list of pivot columns
- generate the final SQL statement based on the input query and the intermediate result,
  adding an AS OF clause to ensure the cursor will not be reused.
*/
create or replace function just_pivot(
  p_INPUT_QUERY in sys.odcivarchar2list,
  p_GROUP_COLS in dbms_tf.columns_t,
  p_PIVOT_COLS in dbms_tf.columns_t,
  p_AGG_COLS in dbms_tf.columns_t,
  p_total_label in dbms_tf.columns_t default null   
) return clob sql_macro is
/*
- PIVOT_IN_LIST concatenates values from all p_PIVOT_COLS columns
  for example, if (DNAME,JOB) then
    ('ACCOUNTING','CLERK') as "ACCOUNTING_CLERK",
    ('ACCOUNTING','MANAGER') as "ACCOUNTING_MANAGER",
  and so on
*/
  l_pivot_in_list_sql long :=  
q'@select listagg('(' || EXPR || ') as "' || AL || '"', ',
') within group(order by #PIVOT_COLS#)
from (
  select distinct
  #EXPR# EXPR,
  #ALIAS# AL,
  #PIVOT_COLS#
  from (#INPUT_QUERY#)
)@';
  l_PIVOT_COLS varchar2(4000);
  l_EXPR varchar2(4000);
  l_ALIAS varchar2(4000);

  l_final_sql long :=
'with pivoted as (
  select * from (
    select #NVL_COLS#,
    #AGG_COLS#
    from (#INPUT_QUERY#)
  )
  pivot(#AGG_MAX_COLS# for (#PIVOT_COLS#) in (
    #PIVOT_IN_LIST#
  ))
)
select * from pivoted
as of scn dbms_flashback.get_system_change_number
order by #ORDER_COLS#';
  l_NVL_COLS varchar2(4000);
  l_AGG_COLS varchar2(4000);
  l_AGG_MAX_COLS varchar2(4000);
  l_PIVOT_IN_LIST varchar2(4000);
  l_ORDER_COLS varchar2(4000);
  
  l_total_label varchar2(32) := 'Total';

begin
  -- set value of l_total_label, which is needed right away
  if p_total_label is not null then
    sqm_util.list_columns(p_total_label, l_total_label, '%s', null, true);
  end if;
  
  -- set values to be plugged into l_pivot_in_list_sql
  sqm_util.list_columns(p_PIVOT_COLS, l_PIVOT_COLS);
  sqm_util.list_columns(
    p_PIVOT_COLS, l_EXPR,
    '''''''''||nvl(%s||null,'''||l_total_label||''')||''''''''', 
    q'§||','||§',
    true
  );
  sqm_util.list_columns(p_PIVOT_COLS, l_ALIAS,
    'nvl(%s||null,'''||l_total_label||''')',
    q'§||'_'||§',
    true
  );
  
  -- plug values into l_pivot_in_list_sql
  l_pivot_in_list_sql := replace(replace(replace(replace(
    l_pivot_in_list_sql,
    '#PIVOT_COLS#', l_PIVOT_COLS),
    '#EXPR#', l_EXPR),
    '#ALIAS#', l_ALIAS),
    '#INPUT_QUERY#', p_INPUT_QUERY(1)
  );
  dbms_output.put_line('line 109');
  
  -- execute l_pivot_in_list_sql
  dbms_output.put_line(l_pivot_in_list_sql);
  execute immediate l_pivot_in_list_sql into l_PIVOT_IN_LIST;
  dbms_output.put_line(l_PIVOT_IN_LIST);
  
  -- set values to be plugged into l_final_sql
  sqm_util.list_columns(
    p_GROUP_COLS multiset union p_PIVOT_COLS,
    l_NVL_COLS,
    'nvl(%s||null, '''||l_total_label||''') %s'
  );

  sqm_util.list_columns(p_AGG_COLS, l_AGG_MAX_COLS, 'max(%s) %s');
  sqm_util.list_columns(p_AGG_COLS, l_AGG_COLS);
  sqm_util.list_columns(p_GROUP_COLS, l_ORDER_COLS,
    'nullif(%s, '''||l_total_label||''')'
  );
  
  -- plug values into l_final_sql
  l_final_sql := replace(replace(replace(replace(replace(replace(replace(
    l_final_sql,
    '#NVL_COLS#', l_NVL_COLS),
    '#AGG_COLS#', l_AGG_COLS),
    '#INPUT_QUERY#', p_INPUT_QUERY(1)),
    '#AGG_MAX_COLS#', l_AGG_MAX_COLS),
    '#PIVOT_COLS#', l_PIVOT_COLS),
    '#ORDER_COLS#', l_ORDER_COLS),
    '#PIVOT_IN_LIST#', l_PIVOT_IN_LIST);
  
  dbms_output.put_line(l_final_sql);
  return l_final_sql;
  
end just_pivot;
/

SQL table macros 8: Print table

I like this one!

Tom Kyte once wrote a procedure to output table contents one column at a time. For some history and a newer solution, see “Print a table with one column name + value per row”.

Using SQL macros, we can get a solution that is even better in my view. My previous solution was a SELECT statement, which I like, but as such it does not hide the code inside a function or procedure, and it is not really configurable.

To stick different column values into one column, we generally have to convert non-string values to VARCHAR2. For fun, I’ll create a little table with several datatypes:

create table datatypes (
C_NUMBER NUMBER,
C_VARCHAR2 VARCHAR2(16),
C_DATE DATE,
C_RAW RAW(16),
C_BINARY_FLOAT BINARY_FLOAT,
C_BINARY_DOUBLE BINARY_DOUBLE,
C_CLOB CLOB,
C_BFILE BFILE,
C_TIMESTAMP TIMESTAMP,
C_TIMESTAMP_WITH_TIME_ZONE TIMESTAMP WITH TIME ZONE,
C_INTERVAL_YEAR_TO_MONTH INTERVAL YEAR TO MONTH,
C_INTERVAL_DAY_TO_SECOND INTERVAL DAY TO SECOND
);

insert into datatypes 
select 3-level,'varchar2',sysdate,hextoraw('FF0102030405'),
  1.1,2.2,'clob',bfilename('DOWNLOAD_DIR', 'x.txt'),
  localtimestamp, localtimestamp,
  interval '1' year, interval '2' day
from dual
connect by level <= 2;

All these datatypes should work from version 19.6 on. If we have access to the Autonomous Cloud 19c version, or to version 21c, we can add some extra datatypes:


create or replace view v_datatypes as
select d.*,
  sys.odcinumberlist(1) c_varray,
  sys.FI_CATEGORICALS('A') c_nested_table,
  SYS.AWRRPT_TEXT_TYPE('a') c_object,
  JSON_object('sysdate':sysdate) j_object
from datatypes d;

Before showing the code of my PRINT_TABLE_MACRO, I’ll show what it does to the view V_DATATYPES. Again, if you try this on a non-cloud 19c version, use the table DATATYPES instead.

select * from print_table_macro(
  p_table => v_datatypes
);
RN COLUMN_NAME COLUMN_VALUE
1 C_NUMBER 2
1 C_VARCHAR2 varchar2
1 C_DATE 2021-12-16T13:40:08
1 C_RAW FF0102030405
1 C_BINARY_FLOAT 1,10000002
1 C_BINARY_DOUBLE 2,2000000000000002
1 C_CLOB clob
1 C_BFILE bfilename(‘DOWNLOAD_DIR’, ‘x.txt’)
1 C_TIMESTAMP 2021-12-16T14:40:8.003345
1 C_TIMESTAMP_WITH_TIME_ZONE 2021-12-16T14:40:8.003345 Europe/Paris
1 C_INTERVAL_YEAR_TO_MONTH +01-00
1 C_INTERVAL_DAY_TO_SECOND +02 00:00:00.000000
1 C_VARRAY [1]
1 C_NESTED_TABLE [“A”]
1 C_OBJECT {“OUTPUT”:”a”}
1 J_OBJECT {“sysdate”:”2021-12-16T13:40:14″}
2 C_NUMBER 1
2 C_VARCHAR2 varchar2
2 C_DATE 2021-12-16T13:40:08
2 C_RAW FF0102030405
2 C_BINARY_FLOAT 1,10000002
2 C_BINARY_DOUBLE 2,2000000000000002
2 C_CLOB clob
2 C_BFILE bfilename(‘DOWNLOAD_DIR’, ‘x.txt’)
2 C_TIMESTAMP 2021-12-16T14:40:8.003345
2 C_TIMESTAMP_WITH_TIME_ZONE 2021-12-16T14:40:8.003345 Europe/Paris
2 C_INTERVAL_YEAR_TO_MONTH +01-00
2 C_INTERVAL_DAY_TO_SECOND +02 00:00:00.000000
2 C_VARRAY [1]
2 C_NESTED_TABLE [“A”]
2 C_OBJECT {“OUTPUT”:”a”}
2 J_OBJECT {“sysdate”:”2021-12-16T13:40:14″}
 

Notice that the input rows are not sorted: I just use ROWNUM to identify each row. Also, each schema-level “User Defined Type” structure is converted to a string in JSON format. Now suppose I want to identify each row by one more columns, and sort by those columns.

select * from print_table_macro(
  p_table => v_datatypes, 
  p_key_cols => columns(c_number)
);
C_NUMBER COLUMN_NAME COLUMN_VALUE
1 C_VARCHAR2 varchar2
1 C_DATE 2021-12-16T13:40:08
1 C_RAW FF0102030405
1 C_BINARY_FLOAT 1,10000002
1 C_BINARY_DOUBLE 2,2000000000000002
1 C_CLOB clob
1 C_BFILE bfilename(‘DOWNLOAD_DIR’, ‘x.txt’)
1 C_TIMESTAMP 2021-12-16T14:40:8.003345
1 C_TIMESTAMP_WITH_TIME_ZONE 2021-12-16T14:40:8.003345 Europe/Paris
1 C_INTERVAL_YEAR_TO_MONTH +01-00
1 C_INTERVAL_DAY_TO_SECOND +02 00:00:00.000000
1 C_VARRAY [1]
1 C_NESTED_TABLE [“A”]
1 C_OBJECT {“OUTPUT”:”a”}
1 J_OBJECT {“sysdate”:”2021-12-16T13:45:50″}
2 C_VARCHAR2 varchar2
2 C_DATE 2021-12-16T13:40:08
2 C_RAW FF0102030405
2 C_BINARY_FLOAT 1,10000002
2 C_BINARY_DOUBLE 2,2000000000000002
2 C_CLOB clob
2 C_BFILE bfilename(‘DOWNLOAD_DIR’, ‘x.txt’)
2 C_TIMESTAMP 2021-12-16T14:40:8.003345
2 C_TIMESTAMP_WITH_TIME_ZONE 2021-12-16T14:40:8.003345 Europe/Paris
2 C_INTERVAL_YEAR_TO_MONTH +01-00
2 C_INTERVAL_DAY_TO_SECOND +02 00:00:00.000000
2 C_VARRAY [1]
2 C_NESTED_TABLE [“A”]
2 C_OBJECT {“OUTPUT”:”a”}
2 J_OBJECT {“sysdate”:”2021-12-16T13:45:50″}
 

Finally, I can exclude columns if I want.

select * from print_table_macro(
  p_table => v_datatypes, 
  p_key_cols => columns(c_number), 
  p_exclude_cols => columns(C_BFILE,C_TIMESTAMP,C_TIMESTAMP_WITH_TIME_ZONE)
);
C_NUMBER COLUMN_NAME COLUMN_VALUE
1 C_VARCHAR2 varchar2
1 C_DATE 2021-12-16T13:40:08
1 C_RAW FF0102030405
1 C_BINARY_FLOAT 1,10000002
1 C_BINARY_DOUBLE 2,2000000000000002
1 C_CLOB clob
1 C_INTERVAL_YEAR_TO_MONTH +01-00
1 C_INTERVAL_DAY_TO_SECOND +02 00:00:00.000000
1 C_VARRAY [1]
1 C_NESTED_TABLE [“A”]
1 C_OBJECT {“OUTPUT”:”a”}
1 J_OBJECT {“sysdate”:”2021-12-16T13:47:36″}
2 C_VARCHAR2 varchar2
2 C_DATE 2021-12-16T13:40:08
2 C_RAW FF0102030405
2 C_BINARY_FLOAT 1,10000002
2 C_BINARY_DOUBLE 2,2000000000000002
2 C_CLOB clob
2 C_INTERVAL_YEAR_TO_MONTH +01-00
2 C_INTERVAL_DAY_TO_SECOND +02 00:00:00.000000
2 C_VARRAY [1]
2 C_NESTED_TABLE [“A”]
2 C_OBJECT {“OUTPUT”:”a”}
2 J_OBJECT {“sysdate”:”2021-12-16T13:47:36″}
 

The code (depends on package SQM_UTIL)

create or replace function print_table_macro(
  p_table in dbms_tf.table_t,
  p_key_cols in dbms_tf.columns_t default null,
  p_exclude_cols in dbms_tf.columns_t default null
) return clob sql_macro is
  l_col_to_strings long;
  l_col_column_names long;
  l_sql long;
  l_key_list varchar2(4000);
  l_order_by varchar2(4000);
  l_exclude_cols dbms_tf.columns_t;
begin
  l_exclude_cols := p_key_cols;
  if l_exclude_cols is null then
    l_exclude_cols := p_exclude_cols;
  else
    if p_exclude_cols is not null then
      l_exclude_cols := l_exclude_cols multiset union p_exclude_cols;
    end if;
  end if;
  sqm_util.col_to_strings(p_table, l_col_to_strings, l_exclude_cols);
  sqm_util.col_column_names(p_table, l_col_column_names, l_exclude_cols);
  if p_key_cols is null then
    l_key_list := 'rownum rn';
    l_order_by := null;
  else
    sqm_util.list_columns(p_key_cols, l_key_list);
    l_order_by := 'order by ' || l_key_list;
  end if;
  l_sql := '
select * from (
  select ' || l_key_list || ',' || l_col_to_strings || ' from p_table
  ' || l_order_by || '
)
unpivot include nulls (column_value for column_name in (' || l_col_column_names || '))
';
  dbms_output.put_line(l_sql);
  return l_sql;
end print_table_macro;
/