Unable to replace paragraph in ODF Text Document

[rsoika]

I need to replace a text fragment of a ODT Text Document with a new text.
My code looks something like this:

	private void replaceODFTextFragment(OdfTextDocument doc, String pattern, String replace)
			throws InvalidNavigationException {
		TextNavigation searchPattern = new TextNavigation(pattern, doc);
		while (searchPattern.hasNext()) {
			logger.info("..found match "+pattern + "    -----> Replace: "+replace);
			TextSelection textSelection = (TextSelection) searchPattern.getCurrentItem();
			textSelection.replaceWith(replace);
			break;
		}

	}

But this is not working.
The OdfTextDocument is not updated at all.

Also it is not possible to navigate to the next element as stated in the documentation [here](https://odftoolkit.org/api/odfdom/org/odftoolkit/odfdom/incubator/search/Navigation.html#getNextMatchElement(org.w3c.dom.Node).

There is no getNextMatchElement method in version 0.11.0 of this project.

Can anybody help me out here.

I use the library successful for OdfSpreadsheetDocument objects and this works well.

[svanteschubert]

Dear Ralph,

Michael is on vacation and I have my head currently wrapped around a different difficult problem, so let me just give you a pointer how I would solve it.

Background

This functionality comes likely from the IBM team that had abandoned ODFDOM, when IBM abandoned OpenOffice (OO). My guess for this abandonment: LibreOffice (LO) could copy new features from OO but OO could not copy features from LO due to copyright.

Basic Approach

If something does not work as expected, create a regression test (best reuse an existing one) and see what failes - narrow it down.

I love to use command-line/bash on Linux to check this.

Go into the test directory and search for "TextNavigation":

/dev/odf/odftoolkit-latest/odfdom/src/test$ find . -name ".java" | xargs grep TextNavigation
./java/org/odftoolkit/odfdom/doc/table/TableRowColumnTest.java:import org.odftoolkit.odfdom.incubator.search.TextNavigation;
./java/org/odftoolkit/odfdom/doc/table/TableRowColumnTest.java: TextNavigation search = new TextNavigation("cell", odtdoc);
./java/org/odftoolkit/odfdom/incubator/search/TextStyleNavigationTest.java: TextNavigation search2;
./java/org/odftoolkit/odfdom/incubator/search/TextStyleNavigationTest.java: TextNavigation search3;
./java/org/odftoolkit/odfdom/incubator/search/TextStyleNavigationTest.java: search2 = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextStyleNavigationTest.java: search3 = new TextNavigation("Roman16 Romanl16delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextStyleNavigationTest.java: search2 = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextStyleNavigationTest.java: search3 = new TextNavigation("deleteRoman16 Romanl16", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextStyleNavigationTest.java: search2 = new TextNavigation("Century22", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: TextNavigation search;
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: TextNavigation nextsearch = new TextNavigation("next", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: TextNavigation search1 = new TextNavigation("change", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("changedelete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: TextNavigation search1 = new TextNavigation("change", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("deletechange", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("ODFDOM", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: TextNavigation nextsearch = new TextNavigation("next", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("Odf Toolkit", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: final List navigations =
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: .map(s -> new TextNavigation(Pattern.compile(s), doc2))
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("^delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextSelectionTest.java: search = new TextNavigation("<%([^>])%>", doc);
./java/org/odftoolkit/odfdom/incubator/search/MONPTest.java: TextNavigation search = new TextNavigation("mnop", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java:/** Test the method of class org.odftoolkit.odfdom.incubator.search.TextNavigation /
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java:public class TextNavigationTest {
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java: private static final Logger LOG = Logger.getLogger(TextNavigationTest.class.getName());
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java: TextNavigation search;
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java: Logger.getLogger(TextNavigationTest.class.getName()).log(Level.SEVERE, e.getMessage(), e);
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java: /* Test getCurrentItem method of org.odftoolkit.odfdom.incubator.search.TextNavigation /
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java: search = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java: /* Test getNextMatchElement method of org.odftoolkit.odfdom.incubator.search.TextNavigation */
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java: search = new TextNavigation("delete", doc);
./java/org/odftoolkit/odfdom/incubator/search/TextNavigationTest.java: Logger.getLogger(TextNavigationTest.class.getName()).log(Level.SEVERE, e.getMessage(), e);

Does one of the tests is similar to your required functionality?
Why those tests not failing?

I hope I could point you into some useful direction, Ralph!

Godspeed and good luck!
Svante

[rsoika]

@svanteschubert thanks for your response. Gives me hope again that I am not alone ;-)
Ok, the package name org.odftoolkit.odfdom.incubator. for the TextNavigation class was already suspicious.
I think I will now follow a different approach using the general methods findFirstChildNode, countChildComponents, findNextChildNode to navigate through the document and find the node of interest. I have seen that it is, of course, possible to replace parts of a document (which is my main goal here).

I will post my results here.....

[svanteschubert]

@rsoika of course you are not alone - instantly reminded me of the song fragment:
'hundred billion bottles washed up on the shore' ;-)

I double checked the JavaDoc problem you mentioned, I guess the docu is right but the access rights might not:
TextNavigation is a subclass of Navigation and getNextMatchElement is only protected at Navigation.

Feel free improving the situation! You help (pull request) is very much welcome... :-)
You might want to discuss design changes ahead to receive quick feed-back... this is what I usually do, when I call Michael or write GitHub issues to sort my thoughts.. ;-)

PS: the 'incubator' package was a bad idea to indicate beta versions, in Google Guava they are using @Beta annotations. Now we are uncertain if we should 'break' the API by dropping those package names... Opinions?

[rsoika]

Yes I verified this. Your are right, it is the wrong protection of the methods.
At the moment it took me to much time and I can't have a deeper look (there were other problems with getting the selection OdfElement.

Maybe I find time the next days to test it further. In the moment I have created an ugly search method in my project to solve the problem somehow. (not the perfect world).

Regarding the incubator package: I would suggest to remove the package and place the code in a official package. Independent from this the problems can be fixed over the time. But with the incubator package other projects (like my own) have more problems by generating source code with need to be refactored too in the future which should not be the case...

[rsoika]

@svanteschubert I created a pull request fixing the issues with the TextNavigation.
The scope of the protected methods was not the reason of the problem. At least I it was simply a missing next() method in the navigator interface, to use the navigator in a way you would expect.
Also there was a minor issue with a OutOfBoundsException which I have also fixed.

See my pull request here: #236

And still think the package name should be changed form org.odftoolkit.odfdom.incubator.search to org.odftoolkit.odfdom.search. Better now than later.

Please let me know what you think.

[svanteschubert]

@rsoika
Cool, Ralph! That was quick!

Although it is a little early for x-mas, but I would love to have a regression test (extended or added) that tests your functionality!
Would be great if this test would fail without your patch and works with - and touch these edge states like indexoutofbound as well! Those regression tests did help a lot to keep sanity over the past years! :-)

Do not want to stretch it, but know you got it in your short term memory and it might be relative quick to produce for you! :-)
If you have a test, I would merge and create a Maven Snapshot release right away (after my review - dis/enable change - see how test reacts, etc.)...

Regarding the "incubator" I am on your side. I think, too, we should remove it,
Perhaps to keep the pain for users low it might make sense to do first another release iteration for ODF 1.2 with the old package name and not breaking their builds (and also right away for ODF 1.3 - only switchting the generated ODF) and then afterwards doing a 1.0 release with the "repaired" name, breaking compatibility but making it somehow more obvious! :-)

[rsoika]

@svanteschubert I added an additional test method to test the behaviour. I also did a lot of testing of other aspects when I wrote my test.
There are already some test methods but all these methods did only test the internal behaviour of the TextNavigator. There was no use case typical to that what a client would do: search for a specific part of a document and change the content.

And changing of the content during the navigating through the document was the part that (in my opinion) never was working correctly.
I tried a lot of refactoring, but at the end this all leads only to new problems, so I left the code as it is - just with my small changes.
The try/catch block which I added to the setCurrentTextAndGetIndex method is in my opinion the only way that works for all the cases. I am not super happy with that.
The main problem is the design of the navigator in its core (which I can't change at this point). In my eyes it should be an Iterator interface

while (nav.hasNext()) {
  result=nav.next();
 ...
}

But this is not the case and the next method was missing. The hasNext method did the navigation, which should be the task of the next method (for my understanding).

For example something like this did not work

while (nav.hasNext()) {
  result=nav.next();
  if (.... && nav.hasNext()) {
     // skip next
     nav.next();
   }
.....
 ...
}

In deed the core implementation has the idea to navigate in a paragraph and does not provide the idea to manipulate a paragrpah and skip to the next one.

But with my changes I think the code is fine and useful.

Let me know what you think about my work....

[svanteschubert]

Thanks again for your fix, Ralph!

I did my part and merged your patch now into the master and have uploaded a Maven SNAPSHOT release.
To not forget your request, I have written myself a follow up issue for the "incubator" package

Have a nice weekend!
Svante

[rsoika]

I wrote a short tutorial about the TextNavigation: https://ralph.blog.imixs.com/2023/06/24/find-and-replace-in-odf-documents/

[rsoika]

@svanteschubert I am sorry to bother you again, but I am still not happy with my last changes of the TextNavigation API.
I introduced the method next() to return the OdfElement containing the selection. This was because I simply missed a next method and the method .getCurrentItem() was in my eyes very confusing. But I am unhappy with this change and I want to ask you to change the methods of the Navigation interface in the following way:

• next() returns the current Selection of the Navigator
• getElement() returns the corresponding ODF element of the current Selection

This would lead to a much more clear API:

	textNav = new TextNavigation("myexpression", odt);
	while (textNav.hasNext()) {

		TextSelection selection = (TextSelection) textNav.next();
		logger.info("...Position=" + selection.getIndex());
		logger.info("...Text=" + selection.getText());

		// fetch paragraph of current selection....
		OdfElement paragraph = textNav.getElement();
		String content=paragraph.getTextContent();
		paragraph.setTextContent("Just a Example");}

	}

[svanteschubert]

@rsoika Not a problem as software is an iterative process ;-) Let me come back to you in the next days, I got a summer flue and have to take some days off..